Python List Mutating while looping over elements

Featured

Tags

, ,

Recently, i faced an issue in my code, which made me realise the value of Immutability, also need to understand how everything works in deep.

Simply, say based on a condition, we need to remove elements in an array (List).
So its simple, loop through the array, if the fail_condition is true, remove the element.

logic to delete the array based on condition

def fail_cond(x): return x==2 or x==3

arr = [1,2,3,4,5,6]

for e in arr:
    print e
    if fail_cond(e): arr.remove(e)
#lets check the original array
print arr

cool, i am expecting the output to be [1,4,5,6]
[Try it out and see the answer yourself, change the fail_cond as you like eg: 1<e<4]

But its not!

It proves that the python, iterates through the array with index, and we are mutating the original array. so it fails to meet our general expectation. it might miss some elements because the array elements positions are changed (like shifting to the left).

original array: [1, 2, 3, 4, 5, 6] then while deleting 2, its next index is 2, so it picks arr[2] which is 4, since we deleted an element : [1, 3, 4, 5, 6]

1 [1, 2, 3, 4, 5, 6]
2 [1, 2, 3, 4, 5, 6]
4 [1, 3, 4, 5, 6]
5 [1, 3, 4, 5, 6]
6 [1, 3, 4, 5, 6]
[1, 3, 4, 5, 6]

So better way is to get a new copy of the array and remove it in original array of using filter, ( but it also depends because i had to do some more logic while looping over elements), or use a comprehension.

 filter(lambda e: fail_cond(e), copy_of_orignal_arry) 

Its reminds me of concurrentModificationException that we get in java 🙂

Question: this leads to another curious question, if its expecting the elements by index, then how come its not throwing array index out of bound exception lets say at the end? (also thinking in perspective of compiler optimizing the variable and puts it out of the loop…)

Maybe its not keeping initial length, but using len(arr) while iteration…

Restarting Node Server With fswatch command

Featured

Tags

, , ,

Recently i came up with situation where i was working with node server file, and i had to restart the node server for every change manually, and i felt that as a discomfort.

Even though there are tools out there like node-supervisor, forver, etc to do the job, I want to do it just with fswatch, because i wanted similar thing when i work in different frameworks or code and language, say for instance trying out something in scala, and i need to run the file everytime. So it will come in handy.

So i started trying out things, and finally came up with a shell script that can be passed to the fswatch with the following command.

	fswatch -o folder_to_watch | xargs -n1 './shell_script.sh'

The shell script is listing the running process with ps, and search for my process with grep.
awk gets the pid and kills the process and finally restart the process.

server_file=server.js
start_node_cmd=&quot;nohup node $server_file &gt;&gt; node.out &amp;&quot;
echo $start_node_cmd

echo 'There is a change in file, restarting node'
ps | grep &quot;[n]ode $server_file$&quot; | awk '
{
	if($1!=&quot;&quot;) {
		print &quot;killing process: &quot;$1;
		system(&quot;kill &quot; $1)
	}
}' 

echo &quot;starting server&quot;
eval &quot;$start_node_cmd&quot;

nohup command & is used to start the process in background and redirecting output to node.out also ignoring SIGHUP signal, nohup by default it redirects output to nohup.out.

The reason i had to use grep ‘[n]ode’ is to exclude the grep process from the process list.

Playing with Python, Scala, Java calculating 100! (Realised the Power of Dynamic Programming)

Featured

Tags

, , , ,

This time, i was playing with product on N numbers (1 to N) factorial with python, scala and obviously the equivalent java code to say functional programming is awesome.

Python:

 res = reduce(lambda x,y: x*y, range(1,101)) 

Scala:

(1 to 100).product

But this time i realised one of the values (i guess there should be a lot!) of (python). I expected the scala code to return the right value (i don’t even know how much is 100! and i thought it’ll give me the right one), but it returned 0! (WOW!!!)

But python on the other hand returned me

93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000L

Ok lets keep it simple, 20! python returned 121645100408832000, but scala returned 0, Nice, because it considers it as Int. and it exceeded the maximum value of Int. which is

Int.MaxValue
Int = 2147483647

Then i thought i would move to BigInt and it finally gave me the right value.
Scala:

print (BigInt(1) to BigInt(100)).product

The amazing Java code to do the same task:

import java.math.BigInteger;
import java.util.Arrays;

public class Calculator {
    public static void main(String[] args) {
        int[] array = Calculator.getArray(1, 100);
        System.out.println(Arrays.toString(array));
        System.out.println(Calculator.getProduct(array));
    }

    private static BigInteger getProduct(int[] array) {
        BigInteger result = BigInteger.ONE;
        for (int number : array) {
            result = result.multiply(BigInteger.valueOf(number));
        }
        return result;
    }

    private static int[] getArray(int start, int end) {
        int[] array = new int[end - (start - 1)];
        int index = 0;
        while (start &lt;= end)
            array[index++] = start++;
        return array;
    }
}

Then in scala, I printed the factorials of the numbers (trying to find out where it’ll break)

def printVals(arg:Int) = {
	def facts(n:Int, acc:Int):Int = {
		if(n &gt; arg) acc
		else {
			var value = acc * n
			println(n+&quot;: &quot;+value)
			facts(n+1, value)
		}
	}
	facts(1,1)
}

printVals(20) gives me the following output,

1: 1
2: 2
3: 6
4: 24
5: 120
6: 720
7: 5040
8: 40320
9: 362880
10: 3628800
11: 39916800
12: 479001600
13: 1932053504
14: 1278945280
15: 2004310016
16: 2004189184
17: -288522240
18: -898433024
19: 109641728
20: -2102132736

Broke at 17 itself. It can only calculate unto 16! after that it overflows.
If i want to add more i need to change the type of acc and function to return Double and it’ll work. for me it felt like i should not care about this.

Python didn’t expect the datatype and it amazingly handled my requests, i tried for 100,1000,100000 it worked, No overflows, it amazingly used dynamic typing and it helped me a lot.

def printFacts(end):
	acc = 1
	for i in range(1,end):
		acc *= i
		print acc

Which ran for most of the inputs.

In static typed languages, i had to think of how far this application can go and i should really design based on the need, (i should not design system with BigInt for numbers ranging less and also i should not use a datatype which results in overflow)

[ I’m using Python 2.7.5 and  Scala 2.11.1 ]

Links:
http://stackoverflow.com/questions/125367/dynamic-type-languages-versus-static-type-languages
https://wiki.python.org/moin/Why%20is%20Python%20a%20dynamic%20language%20and%20also%20a%20strongly%20typed%20language

Question i got:
How far can python handle it? (i tried running it for a while my system got hanged!!!)

if anyone have more to add, or objection please leave a comment.

A gem is not polished without rubbing, nor a man perfected without trials.

Decorator in Python | Tracking method calls

Tags

,

Recently, came up with a good use case for decorators. even though lot of articles says how decorators work. it confuses as the decorator method returns the same method and prints some message.

The usecase i thought was, in my project, whats the top 10 functions, which have  been used extensively. so i can optimize it, clean the code. Also add default documentation for the the functions by taking its parameters.

@cool is used to decorate methods increment and decrement. decrement ‘ll have documentation of parameters also the user defined one.

@cool is equivalent to

 decorated_function = cool(increment)

In the following code, class Meta does the job of keeping track of method call, also adding the params as documentation, also keeping track of maximum argument value.

class Meta:
    method_counts = {}
    max_value = 0

    @staticmethod
    def called(f):
        Meta.method_counts[f.__name__] = Meta.method_counts.get(f.__name__, 0) + 1

    @classmethod
    def get_method_count(cls, f = None):
        return cls.method_counts.get(f)

    @classmethod
    def update_max_val(cls, f, *args):
        if args:
            cls.max_value = max(args[0], cls.max_value)

    @staticmethod
    def get_arguments_as_doc(f):
        doc = "This method is decorated!\nIt takes %d arguments [" % f.func_code.co_argcount
        doc += " , ".join(f.func_code.co_varnames) + "].\n"
        if f.__doc__:
            doc += "User Defined Doc: " + f.__doc__
        return doc

def cool(f):

    def decorated_function(*args):
        # print("You Are Using A Decorated function")
        Meta.called(f)
        Meta.update_max_val(f, args)
        return f(*args)

    decorated_function.__doc__ = Meta.get_arguments_as_doc(f)
    return decorated_function

@cool
def increment(num, inc_val=1):
    return num + inc_val

newly_decorated_function = cool(increment)

@cool
def decrement(num):
    """decrements a number by 1"""
    return num - 1;

increment(1)
increment(20)
increment(3)

print(increment.__doc__)
print(decrement.__doc__)

decrement(0)
decrement(2)

print(Meta.method_counts)

The output of the above script ‘ll be

This method is decorated!
It takes 2 arguments [num , inc_val].

This method is decorated!
It takes 1 arguments [num].
User Defined Doc: decrements a number by 1

{‘decrement’: 2, ‘increment’: 3}

 

I’ve used python2.7. func_code is not available in 3.5 version.

recent versions of python access func_code as method.__code__.co_varnames

Now you can think about what is @staticmethod, @classmethod is doing ?

References:

Hosting openshift bottle app with mongodb

Tags

, , ,

I’ve hosted an app in Openshift, and it was pretty interesting, mainly figuring out how to debug your application. I mean written code, deployed, hit the page. Ooops. 500 Error. Its working absolutely fine in my local.

You can explore web console to create application, add cartridges, and redeploy apps. But i’ll cover the commands, and my own way of doing 🙂

First Install RHC, The Command line interface for OpenShift.

gem install rhc
rhc setup # only for the first time, follow prompts, login to your openshift account

Then lets create an application. for instance i’m taking ‘blog’ as the application name.

rhc create-app blog python-2.6 # uses python cartridge and creates blog application

now lets add the database for our application.

rhc add-cartridge mongodb-2.4 --app blog

Interesting thing was, i’ve used mongo3.2.3 and written my app, but openshift gave only 2.4, so i had to migrate the code to fit to that version.

Now lets clone the repository to our machine, so that we can code 🙂

rhc git-clone blog
cd blog &amp;&amp; git remote -v # To check the remote url

Cool, Now you can write code and push it. or if you want to push existing repository.

git remote add os ssh://url-of-openshift-application.com

and you can push the code. every push triggers the app deployment.
Main thing that was useful for debugging.
You can login to the machine, or tail the logs.

rhc ssh blog # logs in to machine
rhc tail -a blog # alternatively just look at logs

deployed your app 🙂

For reference, i’ve hosted my service bottle app, with mongodb.
https://github.com/dineshkumar-cse/PreferenceSyncService

I’ll go into details of wsgi.py which is used for deployment and mongodb connection in python app later.
 

Psql Horizontal Table Display And Editor

After long search, finally figured out!

I have a table that i want to lookup in psql client (I really don’t like heavy weight tools eg:sqldeverloper, oracle etc.) The problem is table is huge is and its folded and shown, the columns and its record values is hard to map. (eg: from pg_type)

Screen Shot 2015-11-26 at 4.40.44 pm.png

You can enable Extended display as,

\x # This option helpes to display in vertical fashion.
line1:columns:values of Record1
line2:columns:values of Record2

The above also won’t help much if i have too many records.

There’s an Environment Variable available for modifying the table view.

export PAGER=”/usr/bin/less -S”

really does the magic and doesn’t fold lines and shows me the record with horizontal scroll view. Awesome!  (you can add it in your ~/.bashrc or ~/.zshrc file) alternatively you can pass the variable while invoking the psql or other clients:


PAGER="less -S" psql
# -S flag is important as it says less not to fold lines.

man less | grep ' \-S ' -A 4
-S or --chop-long-lines
   Causes lines longer than the screen width to be chopped  rather than folded. That is, the portion of a long line that does not fit in the screen width is not shown. The default is to fold long lines; that is, display the remainder on the next line.

A sample horizontal output from the table:

select * from pg_type;

Screen Shot 2015-11-26 at 4.37.09 pm.png

One More cool thing i found was we can set the editor for psql and we can edit the last query, (its hard to use history and edit in pasted manner)

set PSQL_EDITOR=vi #or whatever editor you like (emacs, sublime,...)
then /e #opens the editor with last query :)

It works both with pgcli and psql. if you wonder what pgcli is, its simlary client for postgres  with additional features like, auto completion of tables, selecting columns, or psql options (\?) based on the context.

http://pgcli.com/features

DevInfo It worked In: Mac, psql (PostgreSQL) version 9.4.4, pgcli Version: 0.19.1

Dynamically Mocking Response Based On Parameters In Unit Tests

Tags

, , , ,

This post we will mock a method based on parameters passed to method at runtime with a realtime example.

lets say for example, I have a list of user email ids, then i am calling a service to find out list of subscribed and unsubscribed users, so the response from the service will contain two lists, List<String> subscribed and List<String> unsubscribed. The process may contains subscribed to different type of email-service or digest or frequency, or the user interest. once we received the response we can send the current newsletter to users in subscribed list. I also have a constraint in which i can send emails to maximum 5 people.

I’ve mocked the response in a way to test my code like if the total number of characters in Emaild is even then its subscribed, if not its unsubscribed.

    Response generateMockedResponse() {
        Response responseMock = new Response();
        for (String user : getUserEmailDump())
            if (user.length() % 2 != 0)
                responseMock.addToUnSubscribedList(user);
            else
                responseMock.addToSubscribedList(user);
        return responseMock;
    }

It sent only mail to right users (users name have chars of even length). and considering limit here as 5, the following test failed. Service sending mail may fail for some user.


    @Test
    public void WrongTestForValidation_shouldOnlySendMaximumNewsletter() {
        when(serviceSpy.filter((Request) Matchers.any(Request.class))).thenReturn(utility.generateMockedResponse(Utility.getUserEmailDump()));

        Response response = utility.generateMockedResponse(Utility.getUserEmailDump());
        response.setTotalSuccessFullRecipients(response.getSubscribedUsers().size());
        when(serviceSpy.sendNewsletter((Request) Matchers.any(Request.class))).thenReturn(response);

        int totalNewslettersSent = sender.send(serviceSpy, utility.getUserEmailDump());
        assertFalse(&amp;quot;Request should not exceed maximum limit&amp;quot;, totalNewslettersSent &amp;gt; MAX_LIMIT_PER_REQUEST);
    }

Because its is returning more than Maximum_Limit which is not possible at all Since we have handle it in code and sending sublist only.

    //NewsletterSender Logic
    if (subscribedUsers.size() &amp;gt; MAX_LIMIT_PER_REQUEST) {
        System.out.println(&amp;quot;Sending newsletter for &amp;quot; + MAX_LIMIT_PER_REQUEST + &amp;quot; subscribed users&amp;quot;);
        Request newsletterSendRequest = new Request(subscribedUsers.subList(0, MAX_LIMIT_PER_REQUEST));
        response = service.sendNewsletter(newsletterSendRequest);
    }

Eventhought it says “Sending newsletter for 5 subscribed users” in console as it’s handled accordingly, the response is an issue. Beacuse while mocking returning the response as same as total subscribers available, which i have generated is wrong.

The issue is its not dynamic mocking. Its mocking the data, but its using static data, its not based on the new request which. The mock should consider the list (dynamically, i.e the sublist which comes in request and filling the successfulRecipients accordingly)

This made me realize the value of ThenAnswer in mock.

when(serviceSpy.sendNewsletter((Request) Matchers.any(Request.class))).thenAnswer(new Answer&amp;lt;Object&amp;gt;() {
            @Override
            public Object answer(InvocationOnMock invocation) throws Throwable {
                Request request = (Request) invocation.getArguments()[0];
                //whatever condition that makes sending fail
                Iterator&amp;lt;String&amp;gt; subscriberList = request.getEmailList().iterator();
                int sentCount = 0;
                while (subscriberList.hasNext()) {
                    String user = subscriberList.next();
                    if (!user.matches(&amp;quot;.*\\d+.*&amp;quot;)) {
                        sentCount++;
                    } else {
                        System.out.println(&amp;quot;super this is removed&amp;quot; + user);
                    }
                }
                Response response = new Response();
                response.setTotalSuccessFullRecipients(sentCount);
                return response;
            }
        });    

In development or Testing, we don’t use the real services and we prefer mocks / stubs. because the of the cost per service request, or in order to reduce the test time, or the real service might not be available.

Have a look at full code in my git repository : Playing-With-Unit-Test

 

ReduceLeft vs ReduceRight in Scala

Tags

, ,

I thought that reduceLeft and reduceRight just differs in order of traversing the list, like iterating it from left to right and right to left respectively and doing an operation (since i was thinking from iterating array from coding perspective) but i realised its not.

Addition and multiplication didn’t give me any clue as it holds The Associative Property.

(numbers.reduceLeft(_ * _) == numbers.reduceRight(_ * _)) ==
    (numbers.reduceLeft(_ + _) == numbers.reduceRight(_ + _))

So i was wondering where we really need the distinction, trying to think of situation where i really need reduceRight and not reduceLeft in our application, (thinking that i can reverse the array and do reduceLeft and its same)

The following condition

wrongReduceRight(numbers) == reduceLeft(reverse(numbers))

is true in java for my wrongly assumed code and in scala it doesn’t hold.
so its clear that is different than what i initially thought.

reduceLeft in Java:

/*evaluated to -13 correctly*/
 int[] numbers = {1, 2, 3, 4, 5};
 int total = numbers[0];
 for (int index = 1; index &lt; numbers.length; index++) {
 total -= numbers[index];
 }

reduceRight in Java:

/*wrong reduceRight*/
        int total = numbers[numbers.length - 1];
        for (int index = numbers.length - 2; index &gt;= 0; index--) {
            total -= numbers[index];
        }

But thats not the case as the code in java evaluated to -5 but in scala it’s 3.
The reason being is that scala take the last and pair and evaluates from right to left by substituting the values.

reduceRight is evaluated like

(1 – (2 – (3 – (4 – 5))))

(1 – (2 – (3 – (4 – 5))))
(1 – (2 – (3 – (-1))))
(1 – (2 – (4)))
(1 – (-2))
(3)

The code in java i’ve created for the reduceRight

    private static int reduceRight(int[] numbers) {
        int result = 0;
        if (numbers.length &lt; 2) {
            return numbers[0];
        }
        if (numbers.length &gt;= 2)
            result = numbers[numbers.length - 2] - numbers[numbers.length - 1];

        for (int index = numbers.length - 3; index &gt;= 0; index--) {
            System.out.print(numbers[index] + &quot; with &quot; + result + &quot;; &quot;);
            result = numbers[index] - result;
        }
        return result;
    }

reduceRight(numbers); //now evaluated to 3 correctly.

scala code:

numbers.reduceLeft(_ - _)  //evaluates to -13
numbers.reduceRight(_ - _) // evaluates to 3

for more understaing of scala’s _ in  in reduceLeft: http://www.codecommit.com/blog/scala/quick-explanation-of-scalas-syntax

Scala vs Java (Concise)

Tags

, , , ,

I’ve tried to create a program which’s having a map (key, value) pairs that contains the occurrences of the moves, I’ve fixed the moves R, P, S (Thinking of Rock, Paper, Scissor Game).

Every time when i make a move the map should increment the value of the move accordingly.

I’ve tried it with java and scala

This is the java code

import java.util.HashMap;
import java.util.Map;

class Game {

    public HashMap<String, Integer> movesCount = new HashMap<String, Integer>();

    public Game() {
        movesCount.put("R", 0);
        movesCount.put("P", 0);
        movesCount.put("S", 0);
    }


    public static void main(String[] args) {
        Game game = new Game();
        game.makeMove("R");
        System.out.println(game.getFrequentlyUsedMove());
        game.makeMove("S");
        game.makeMove("S");
        System.out.println(game.getFrequentlyUsedMove());
        game.displayMoveOccurrences();
    }

    public void makeMove(String move) {
        Integer existingCount = movesCount.get(move);
        movesCount.put(move, ++existingCount);
    }

    public void displayMoveOccurrences() {
        System.out.println(movesCount);
    }


    public String getFrequentlyUsedMove() {
        int maxCount = -1;
        String frequentMove = null;
        for (Map.Entry<String, Integer> currMove : movesCount.entrySet()) {
            Integer occurrence = currMove.getValue();
            if (occurrence > maxCount) {
                maxCount = occurrence;
                frequentMove = currMove.getKey();
            }
        }
        return frequentMove;
    }

}

and the scala code

val moves_count = scala.collection.mutable.Map("R"->0, "P"->0, "S"->0)
def makeMove(move:String) = moves_count(move) += 1
def getFrequentMove = moves_count.maxBy(_._2)._1

makeMove("R")
println("Far more Cooool!" + getFrequentMove)

makeMove("S")
makeMove("S")
println("Far more Cooool!" + getFrequentMove)

println(moves_count)

This is really amazing, and thats why people say its CONCISE.
scala program is 9 lines but java took 38 lines.

you might think that we can use a sorted Map “TreeMap” and get the value simply, but considering that this method is used rarely sortedMap’ll cause overhead of O(log(n)) complexity.

[TreeMap is implemented as RedBlackTree hence insertion takes O(log n) complexity]

instead of the displayMoveOccurences we might wanna make the moves public and can display it directly so as to reduce number of lines, but i’ve written the program as java developer mindset [how i’ll write in java].

TreeMap complexity:
http://stackoverflow.com/questions/8297223/working-of-tree-maps-time-complexity-at-insert-and-read-opertation-in-java

Python Script to Deploy the Application in Weblogic

Tags

, ,

Using jdeveloper to host the application is painful. It abstracts lot of things. when we try to build and deploy our application by our own code, we will come to  know lot of stuffs happening behind the scenes.

Start the web logic server (.sh file)

WEBLOGIC_DOMAIN_HOME/bin/startWebLogic.sh

we need to set the web logic environment, it sets few variables like WL_HOME, JAVA_HOME, CLASSPATH, PATH, etc.

. $WEBLOGIC_HOME/bin/setWLSEnv.sh
java weblogic.WLST ./weblogicdeploy.py

The following is the python code to deploy our application in web logic server.

import re
warPath = 'WebService/target/build/dist/webservice.ear'
appName = 'webservice'
weblogicUrl = 't3://127.0.0.1:7101'  //change the port number depending on our weblogic server configuration
userName = 'weblogic'
password = 'weblogic1'
connect(userName, password, weblogicUrl)
appList = re.findall(appName, ls('/AppDeployments'))
if len(appList) >= 1:
print 'undeploying application'
undeploy(appName)
deploy(appName, path = warPath, retireTimeout = -1, upload = 'True')
exit()

you can use a war if it contain simple java classes and libs, ear contains same with wars inside (can have multiple applications inside it).

Difference between ear, war, jar:
http://stackoverflow.com/questions/1594667/war-vs-ear-file
Ear Structure:
http://middlewaremagic.com/weblogic/wp-content/uploads/2010/06/EAR_Application_Diagram.jpg

My git repository https://github.com/dineshkumar-cse/Team-Amazing which contains the scrips and application.