Posts

Showing posts from 2017

Programming with Apache Spark and Cassandra -draft

Putting the knowledge gained so far in this  and frequent questions that many may ask and what we have asked ourselves. 1.1          What is the need for using Spark ? Spark gives you horizontal scale ability in a programmer friendly way. 1.2          But what about other options ? There are other options as well. I have listed them below, which describes and highlights Sparks place in the architecture Type Level of Granularity Descritpion LoadBalancer (nginx,haproxy) Request Level (usually HTTP requests) Works well for Request-response type client server protocols. Works also well in context of microservices in application program side However to scale the processing insdie the application programs this is inadequate Task Managers (celery, other MQ based) Task Level Helps to scale processing in the application program.Takes care of Task handling. However the onus is on the developer

Best practises - Selenium WebDriver/ Java

Intermittent failure in Selenium Test cases ? After clicking the drill down sometimes web-elements are not found causing all summary table test cases to fail. Common Root Causes 1)  Prefer Selection By.ID className  then By.cssSelectior and only if all else fails use By.Xpath Selection by CSS  By.CSSSelector  should be preferred over XPath as this is more stable as it is natively supported by browser . XPAth is an abstraction provided by Selenium and not as performant. If XPath is used make sure that you have hand written the XPath and it is performant and not generic in that it has to to brute force search through the entire DOM to find your element. Where ever possible , use By.ID, else By.CSSSelector and in case of no other option use XPath after proper testing. You can use FireFinder plugin for FireFox (first add FireBug) to test your CSS or Xpath (if there is no way you can select by CSS) For example this XPath to finding the drill down element  //*[@id='scTab

Python Profiling - Some hints

No time to compose fully; Here are some links which helped me in CPU profiling Python Profilers http://www.vrplumber.com/programming/runsnakerun/  python -m cProfile -o profile_out ANR_4G_IRAT.py  pacman -S kdesdk-kcachegrind  http://thirld.com/blog/2014/11/30/visualizing-the-results-of-profiling-python-code/  runsnakerun http://wiki.wxpython.org/How%20to%20install%20wxPython  pstats dechipering cprofile output  https://pymotw.com/2/profile/  https://docs.python.org/2/library/profile.html  yappi https://code.google.com/archive/p/yappi/  pyprof2calltree -k -i myscript.cprof  https://julien.danjou.info/blog/2015/guide-to-python-profiling-cprofile-concrete-case-carbonara  performance tips python  https://wiki.python.org/moin/PythonSpeed/PerformanceTips#Profiling_Code