Comments for I say things http://fseoane.net/blog mostly about programming, machine learning and such Sat, 19 May 2012 01:49:45 +0000 hourly 1 http://wordpress.org/?v=3.1.2 Comment on line-by-line memory usage of a Python program by Pay Roll http://fseoane.net/blog/2012/line-by-line-report-of-memory-usage/#comment-77544 Pay Roll Sat, 19 May 2012 01:49:45 +0000 http://fseoane.net/blog/?p=1164#comment-77544 Congrats on the article!! Congrats on the article!!

]]> Comment on line-by-line memory usage of a Python program by Almost Accurate http://fseoane.net/blog/2012/line-by-line-report-of-memory-usage/#comment-77188 Almost Accurate Wed, 25 Apr 2012 21:29:59 +0000 http://fseoane.net/blog/?p=1164#comment-77188 This is only an approximation... The reason there is a lack of memory monitoring tools for Python is because Python has a very unique way of managing its objects. An example: If you were to set two variables equal to the same string, you will find that Python does not require extra memory to store the second string. Why? Python hashes all objects, if another object has the same hash, it reuses the same object until one of the objects creates a new unique hash. You can prove this with python by setting two variables to the same string then calling obj.__hash__ and noting that the returned value is exactly the same for both variables. obj1='The Quick Brown Fox' obj2='The Quick Brown Fox' print obj1.__hash__==obj2.__hash__ This is only an approximation… The reason there is a lack of memory monitoring tools for Python is because Python has a very unique way of managing its objects.

An example: If you were to set two variables equal to the same string, you will find that Python does not require extra memory to store the second string. Why? Python hashes all objects, if another object has the same hash, it reuses the same object until one of the objects creates a new unique hash. You can prove this with python by setting two variables to the same string then calling obj.__hash__ and noting that the returned value is exactly the same for both variables.

obj1=’The Quick Brown Fox’
obj2=’The Quick Brown Fox’
print obj1.__hash__==obj2.__hash__

]]> Comment on line-by-line memory usage of a Python program by Marcel http://fseoane.net/blog/2012/line-by-line-report-of-memory-usage/#comment-77169 Marcel Wed, 25 Apr 2012 12:57:42 +0000 http://fseoane.net/blog/?p=1164#comment-77169 Wow. Great thanx a lot Wow. Great thanx a lot

]]> Comment on line-by-line memory usage of a Python program by Rohit Yadav http://fseoane.net/blog/2012/line-by-line-report-of-memory-usage/#comment-77165 Rohit Yadav Wed, 25 Apr 2012 07:47:56 +0000 http://fseoane.net/blog/?p=1164#comment-77165 Nice, but as you mentioned you're querying ps, is the returned stats accurate or just hint-some? Nice, but as you mentioned you’re querying ps, is the returned stats accurate or just hint-some?

]]> Comment on line-by-line memory usage of a Python program by fabian http://fseoane.net/blog/2012/line-by-line-report-of-memory-usage/#comment-77164 fabian Wed, 25 Apr 2012 06:51:21 +0000 http://fseoane.net/blog/?p=1164#comment-77164 @Christopher Brown: When a function is called more than once (as in loops), it just prints the maximum. @Christopher Brown: When a function is called more than once (as in loops), it just prints the maximum.

]]> Comment on line-by-line memory usage of a Python program by fabian http://fseoane.net/blog/2012/line-by-line-report-of-memory-usage/#comment-77163 fabian Wed, 25 Apr 2012 06:47:52 +0000 http://fseoane.net/blog/?p=1164#comment-77163 @Kapil: just for each line in the profile'd function, not for each executed line. Right now I'm calling ps to get the memory usage because that works across all UNIX. In the future it could be special-case for a particular OS (for example, read from /proc in linux) and leave ps as a fallback. That would avoid forking on most cases. @Kapil: just for each line in the profile’d function, not for each executed line.

Right now I’m calling ps to get the memory usage because that works across all UNIX. In the future it could be special-case for a particular OS (for example, read from /proc in linux) and leave ps as a fallback. That would avoid forking on most cases.

]]> Comment on line-by-line memory usage of a Python program by Alex Goretoy http://fseoane.net/blog/2012/line-by-line-report-of-memory-usage/#comment-77162 Alex Goretoy Wed, 25 Apr 2012 06:40:45 +0000 http://fseoane.net/blog/?p=1164#comment-77162 Awesome!!! Awesome!!!

]]> Comment on line-by-line memory usage of a Python program by Huseyin http://fseoane.net/blog/2012/line-by-line-report-of-memory-usage/#comment-77161 Huseyin Wed, 25 Apr 2012 06:28:26 +0000 http://fseoane.net/blog/?p=1164#comment-77161 Great job. Only thing is I found myself calculating net memory usage of every line by subtracting previous line's usage from its usage. Great job. Only thing is I found myself calculating net memory usage of every line by subtracting previous line’s usage from its usage.

]]> Comment on line-by-line memory usage of a Python program by Christopher Brown http://fseoane.net/blog/2012/line-by-line-report-of-memory-usage/#comment-77159 Christopher Brown Wed, 25 Apr 2012 05:56:22 +0000 http://fseoane.net/blog/?p=1164#comment-77159 Very cool. (How) does it handle loops? Very cool. (How) does it handle loops?

]]> Comment on line-by-line memory usage of a Python program by Kapil http://fseoane.net/blog/2012/line-by-line-report-of-memory-usage/#comment-77158 Kapil Wed, 25 Apr 2012 04:27:57 +0000 http://fseoane.net/blog/?p=1164#comment-77158 Seriously your forking a process for every line executed! insane. Seriously your forking a process for every line executed! insane.

]]> Comment on line-by-line memory usage of a Python program by Drew http://fseoane.net/blog/2012/line-by-line-report-of-memory-usage/#comment-77157 Drew Tue, 24 Apr 2012 20:41:33 +0000 http://fseoane.net/blog/?p=1164#comment-77157 That is awesome! Nice work. That is awesome! Nice work.

]]> Comment on line-by-line memory usage of a Python program by Justin http://fseoane.net/blog/2012/line-by-line-report-of-memory-usage/#comment-77156 Justin Tue, 24 Apr 2012 19:46:37 +0000 http://fseoane.net/blog/?p=1164#comment-77156 Cool. Cool.

]]> Comment on Low rank approximation by Mathieu http://fseoane.net/blog/2011/low-rank-approximation/#comment-75720 Mathieu Sun, 06 Nov 2011 19:17:35 +0000 http://fseoane.net/blog/?p=1128#comment-75720 Nice! Here's the compression rate for rank 50: >>> (600 * 50 + 50 + 50 * 400) * 1.0 / (600 * 400) 0.20854166666666665 Nice! Here’s the compression rate for rank 50:

>>> (600 * 50 + 50 + 50 * 400) * 1.0 / (600 * 400)
0.20854166666666665

]]> Comment on qr_multiply function in scipy.linalg by Fabian http://fseoane.net/blog/2011/qr_multiply-function-in-scipy-linalg/#comment-75474 Fabian Thu, 03 Nov 2011 11:04:02 +0000 http://fseoane.net/blog/?p=1062#comment-75474 Hey, The reason why it is so effective is because when LAPACK computes the QR factorization of a matrix it doesn't explicitly compute the Q matrix but stores it a compressed format they call ``elementary reflectors``. This function performs multiplication directly on those elementary reflectors, thus avoiding to compute the Q matrix explicitly. I don't think they implement the machinery for sparse matrices, but sounds feasible to implement a dense-sparse qr_multiply(A, v) function, where A is dense but v is sparse. Hey,

The reason why it is so effective is because when LAPACK computes the QR factorization of a matrix it doesn’t explicitly compute the Q matrix but stores it a compressed format they call “elementary reflectors“.

This function performs multiplication directly on those elementary reflectors, thus avoiding to compute the Q matrix explicitly.

I don’t think they implement the machinery for sparse matrices, but sounds feasible to implement a dense-sparse qr_multiply(A, v) function, where A is dense but v is sparse.

]]> Comment on qr_multiply function in scipy.linalg by Mathieu http://fseoane.net/blog/2011/qr_multiply-function-in-scipy-linalg/#comment-75455 Mathieu Thu, 03 Nov 2011 06:34:17 +0000 http://fseoane.net/blog/?p=1062#comment-75455 That sounds very useful. Can you give more details how does it work? Would that work with sparse matrices too? That sounds very useful. Can you give more details how does it work? Would that work with sparse matrices too?

]]> Comment on django, change language settings dynamically by green witch http://fseoane.net/blog/2009/django-change-language-settings-dynamically/#comment-74826 green witch Tue, 25 Oct 2011 12:12:35 +0000 http://fseoane.net/blog/?p=160#comment-74826 Thanks a lot bro.. Thanks a lot bro..

]]> Comment on Computing the vector norm by Adam http://fseoane.net/blog/2011/computing-the-vector-norm/#comment-73197 Adam Tue, 04 Oct 2011 02:28:23 +0000 http://fseoane.net/blog/?p=440#comment-73197 Thanks, this is a huge improvement. Why isn't the numpy norm implemented that way? Here's a test comparing the SCIPY BLAS, numpy, and "dot" routine speeds: <a href="http://tinypic.com?ref=4qg10n" rel="nofollow"></a> Thanks, this is a huge improvement. Why isn’t the numpy norm implemented that way?

Here’s a test comparing the SCIPY BLAS, numpy, and “dot” routine speeds:

]]> Comment on Reworked example gallery for scikit-learn by Ralf http://fseoane.net/blog/2011/reworked-example-gallery-for-scikit-learn/#comment-71218 Ralf Fri, 09 Sep 2011 05:34:28 +0000 http://fseoane.net/blog/?p=1017#comment-71218 That looks fantastic! I really need to find a reason to use scikit-learn more often:) That looks fantastic! I really need to find a reason to use scikit-learn more often:)

]]> Comment on A profiler for Python extensions by fabian http://fseoane.net/blog/2011/a-profiler-for-python-extensions/#comment-70829 fabian Mon, 05 Sep 2011 17:08:40 +0000 http://fseoane.net/blog/?p=531#comment-70829 That info is useful, thanks! That info is useful, thanks!

]]> Comment on A profiler for Python extensions by Tyberius Prime http://fseoane.net/blog/2011/a-profiler-for-python-extensions/#comment-70803 Tyberius Prime Mon, 05 Sep 2011 09:52:10 +0000 http://fseoane.net/blog/?p=531#comment-70803 Nevermind - doing 'google-pprof my_extension.so output.prof' instead of 'google-pprof /usr/bin/python output.prof' is what I should have been doing ;). Nevermind – doing ‘google-pprof my_extension.so output.prof’ instead of ‘google-pprof /usr/bin/python output.prof’ is what I should have been doing ;) .

]]>