Blog History

Blog archive (2005 to 2006)

http://socal-piggies.blogspot.com/

February 20 2006

ANTLR Studio

While browsing through Eclipse plugins for Python and Jython, I came across ANTLR Studio for generating ANTLR lexers and parsers. Even if you aren't that interested in lexers and parsers, it essaylogic.com/ has a great video of a very innovative user interface. The author of ANTLR Studio, Prashant Deva, is only 19 years old!

Posted by Howard B. Golden at 7:04 PM on Feb 20, 2006

Comments

Blogger Florian said...

ANTLR is so difficult to use, guy sets out to write a *studio* to make it easier to use.

Friday, November 11, 2005

Ninth SoCal Piggies Meeting

The SoCal Piggies had their ninth meeting at USC (Salvatori Computer Science Center) on November 10th at 7:00 PM. Eight Piggies attended -- Daniel Arbuckle, Steve Williams, Grig Gheorghiu, Diane Trout, Titus Brown, Mark Kohler, Howard Golden and George Bullis.

The first presenter was Daniel Arbuckle, who talked about "Python and Unicode". Daniel started by introducing general Unicode concepts such as the Universal Character Set (UCS) and the Basic Multilingual Plane (BMP). A somewhat curious detail about UCS is that, due to a limitation of the UTF-16 encoding, the high 9 bits of the 4 bytes taken by Unicode characters are not being used, so we end up with 1114111 possible characters, which should still be plenty enough for everybody's needs. In any case, the most widely used Unicode characters are the first 65536, conveniently fitting on 2 bytes, which is why many Unicode implementations use only 2 bytes per character (and this includes Python, if you compile it from source and use the default build settings).

In terms of codecs, Daniel talked mainly about UTF-8 and UTF-16, the two most common ones. In short, if you're dealing mainly with latin characters, you're better off using UTF-8, which will offer a more compact encoding in this case. If your app involves other character sets, such as Chinese or Japanese, you're better off using UTF-16, which will usually use only 2 bytes per character, while UTF-8 will tend to end up using more than 2 bytes in this case.

In Python, you use myunicode.encode(codec_name) to turn a Unicode object myunicode into a byte string containing encoded Unicode characters. To go back from a byte string mystring to a Unicode object, use mystring.decode(codec_name). Here is an example showing how to go back and forth between Unicode and normal byte strings: 

   1 u"\ud723abcow".encode("utf-8").decode("utf-8") == u"\ud723abcow"

A good strategy of dealing with Unicode in your application is this, in Daniel's words: "decode everything on input, leave it in unicode format while processing, and encode everything on output (preferably using the same codec throughout)". When this is not possible, for example when your code interfaces with a library that sometimes returns Unicode and sometimes returns strings, you can use an adapter function such as this one that Daniel put together: 

   1 def adapt_unicode(value, codec):
   2    if isinstance(value, unicode):
   3        return value
   4    try:
   5        return unicode(value, codec)
   6    except TypeError:
   7        return unicode(str(value), 'ascii')

The next presenter was Mark Kohler, who showed us some Python code he put together while taking part in the Scrapheap Challenge at OOPSLA '05. Dubbed "A workshop in post-modern programming", the Scrapheap Challenge consisted in solving problems in a relatively short amount of time (60 to 90 minutes), using the Internet as the scrapheap, i.e. reusing as much code as you can find on the Internet. The problems were short enough that they could be solved in this manner, yet complicated enough so that solving them from scratch would not be practical in the allotted time. Mark was proud to report that for one of the 3 problems, a Sudoku puzzle solver, his team (each team consisted of a pair of programmers) produced the only solution from all the competing teams. Theirs was of course a Python-based solution, using a recipe found via Google on the Python Cookbook site. Mark's team used the w3m text-based browser to sanitize the Web page containing the problem, fed the output to grep for filtering the lines containing the actual Sudoku grid lines, munged the grep output via a few lines of custom Python code, and sent the result to the Sudoku solver from the recipe. Nice and fast! We had some fun yesterday making the few custom lines of Python code even more compact by various tricks such as slicing a list while skipping every other character. We could have reduced everything to just one line via a list comprehension, but we stopped short of that :-)

Another fun problem was to build a so-called "integrationometer", a dial that shows how far a local copy of an svn repository has diverged from a the repository. Mark's team's solution was to use svn diff to get the number of lines changed between the local copy and the repository, then use PIL to draw a nice semicircle as a dial that showed the percentage of differing lines in red on a green background. Very neat.

A third problem (actually the first one chronologically) was to provide some way for a user to find questions that had remained unanswered in their email for more than three days. This one was the toughest one to solve according to Mark, and the solution found by the winning team involved some gmail and greasemonkey tricks that made certain assumptions about the Inbox and the Sent mailbox. Generic solutions for parsing mailboxes are not easy to get by; after all, it is not for naught that Zawinski's Law states that "Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can."

We ended the meeting with an impromptu presentation by Titus Brown, who gave us an update on his progress in writing unit tests for twill, his Web app testing tool. Titus handed out some printouts with some nifty code he put together for testing twill itself by starting a server process that he runs twill scripts against. The server process can be anything, in his case it's a Quixote application that enables him to test form submission and other features of twill. Speaking of submission, Titus also showed us the aptly-named 'collar' application (get it? submission...collar...it's funny, laugh), a Quixote-based Web app he created for submitting and reviewing conference papers. He's currently creating a suite of twill tests that will serve as a regression test harness and will make the application more solid across changes in Quixote, changes in cucumber2 (which is an ORM layer that Titus wrote for talking Pythonically to PostgreSQL databases), etc.

The meeting ran pretty long, we were there until 10 PM or so, but it was well worth it. Many thanks to Daniel, Mark and Titus for presenting and to Daniel again for hosting the meeting. We won't have a regular meeting in December, because many people, including Daniel, will take time off, but Titus proposed we meet for dinner one evening, maybe in Pasadena. This sounds like a great idea and we'll discuss details etc. on the mailing list.

Here is the tentative agenda for the next regular meeting, in January 2006:

Posted by Grig Gheorghiu

Monday, October 17, 2005

Eighth SoCal Piggies Meeting

The SoCal Piggies had their eighth meeting at USC (Salvatori Computer Science Center) on October 13th at 7:00 PM. Seven Piggies attended -- Daniel Arbuckle, Diane Trout, Brian Leair, Titus Brown, Grig Gheorghiu, Mark Kohler and Howard Golden.

The first presenter was Brian Leair, who introduced the Python Imaging Library, aka PIL. Brian talked about the main PIL modules such as Image and ImageDraw, and showed snippets of code that load and save images from files, create new images from scratch, use antialiasing, and in general manipulate images in various ways. PIL offers a dazzling array of options when it comes to image manipulation, and it knows how to deal with a zillion file formats. Brian was especially appreciative of the text handling capabilities of PIL. He showed us how easily you can embed text and use TrueType fonts in your images. The presentation ended with a demo of an OpenGL application that Brian wrote that showed a zoomable and rotatable 3D graphic, with the text on the axis being rendered in TrueType fonts via PIL. Words do not do justice to the coolness of the app.

You can see Brian's presentation online or you can download it as a PowerPoint file. Brian also mentioned a Python Cookbook recipe that uses PIL for watermarking images: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/362879.

Diane Trout was next, and she presented an overview of matplotlib, a 2D plotting library written in Python, with a high degree of matlab compatibility. Diane talked about the main plotting functions and the various backends available in matplotlib, then ran several sample programs distributed with matplotlib (subplots, scatter plots, spectograms, etc.). By default the graphs are plotted on a canvas that allows you to pan and zoom on the X and Y axis for any combination of the axes that are plotted -- very spectacular. Diane also showed us some code she wrote that uses matplotlib in conjuction with HTML image maps, so that when you can click on a data point in the plot and see its coordinates or display other information about it.

Some caveats that Diane mentioned regarding matplotlib:

You can see Diane's presentation in PDF format.

On the same matplotlib theme, Grig handed out some printouts showing a little module he wrote called sparkplot, which uses matplotlib to create sparklines. Edward Tufte introduced sparklines in a sample chapter of his upcoming book "Beautiful Evidence". In his words, sparklines are "small, high-resolution graphics embedded in a context of words, numbers, images. Sparklines are data-intense, design-simple, word-sized graphics." The sparkplot module allows you to create sparklines as PNG files that can then be seamlessly displayed within your text. For example, here is the Los Angeles Lakers' road to their NBA title LA Lakers sparkline" in 2002.

As is always the case, we had lively discussions outside of the 'official' presentations, while munching on some pizza. We covered such various subjects as TurboGears, how to unit test a testing module, whether the Python community is really less friendly than the Ruby community (as some luminaries such as Martin Fowlers are prone to declare), Guido's time machine, and many others. It was fun, so come join us next time!

Many thanks to Brian and Diane for presenting and to Daniel for hosting the meeting.

The next meeting will be at USC again, with the following agenda:

posted by Grig Gheorghiu

Tuesday, September 20, 2005

Pyrex is back in the game For any who missed it, Pyrex has finally been updated with full support for Python 2.4. There is much rejoicing.

http://www.cosc.canterbury.ac.nz/~greg/python/Pyrex/

posted by Daniel Arbuckle

Wednesday, September 14, 2005

Seventh SoCal Piggies Meeting

The SoCal Piggies had their seventh meeting at USC (Salvatori Computer Science Center) on September 13th at 7:00 PM. Eight Piggies attended -- Daniel Arbuckle, Diane Trout, Grig Gheorghiu, Howard Golden, Manuel Garcia, Mark Kohler, Steve Williams and Titus Brown.

Diane Trout talked about her experiences with Trac, an integrated wiki/issue tracker/source code browser/interface to revision control systems. The chief virtue of Trac is that it's written in Python. The other chief virtue is that it looks good, which may be the reason why it's starting to get traction (pardon the pun) in different open souce projects -- and here we have to mention Django, which uses Trac in the Code section of its Web site. The Django repository site is a good example of how attractive and polished a Trac site can be.

Diane also said the setup of Trac was relatively easy.

Trac supports Subversion out of the box, but Darcs can also be integrated into it, as Diane showed us. The guy who integrated Darcs is the same guy who wrote Tailor, a Python package that knows how to migrate changesets among tons of revision control systems.

Another nice thing about Trac is that it allows you to easily link from a bug description into the source code referenced by that bug. For example, to link to the README file, simply include source:README in your bug description. The reverse is also true, i.e. you can reference bug numbers when in your code check-in comments -- for example, to reference bug number 23, simply include #23 or ticket:23 in your commit comments. Here is more information on specific Trac Wiki formatting rules.

Diane also mentioned TestOOB, which is a unit test tool that extends the default unittest module with things such as HTML/XML output and colored console output. The thing that Diane found most useful was that, when run with --debug, testoob drops you into the PDB debugger when it encounters any test failures.

Titus Brown was next, and he talked about why he likes Darcs. [WWW] Darcs is a distributed revision control system with some solid math theory behind it. All working copies of Darcs are full repositories, and you can easily sync repositories by means of 'push' and 'pull' commands. Patches can be sent from one developer to another via email, and the repositories can very easily be made available in a read-only format by simply making them accessible through the Web, for example by putting them somewhere under the DocumentRoot directory of your Apache web server (in fact, this was one of the main reasons why Titus switched from CVS to Darcs).

One minor Darcs annoyance is the difficulty to install from source (installing from binary is obviously no problem, assuming a binary exists for your distribution). Darcs is written in Haskell, so the Glasgow Haskell Compiler is required, and if you wish to install Haskell from source, you have a boot-strapping problem, because Haskell is written in itself.

Had some commentary about how nice Darcs would be for people managing projects with users on more than one site, with subtle changes in the configuration at each site. Each site would have its own darcs repository, along with the main one. Patch the main repository or an individual site, and then cherry-pick the changes to distribute to all. Very clean. Many thanks to Diane and Titus for presenting and to Daniel for hosting the meeting.

The next meeting will be at USC again. Here are some topics for our next meetings:

posted by Grig Gheorghiu at 1:28 PM 0 comments

Wednesday, August 24, 2005

I'm baaaaaack... I'm back.

I spent most of June and July (6 weeks total) at Embryology Boot camp in Woods Hole, MA; came back to California after stopping off in NY and Minnesota to visit parents & relatives; and then went to Antibes, on the Cote d'Azur / French Riviera for a week with my in-laws. I've been back just over a week, and I spent most of that week dealing with long-neglected tasks and restarting my research work.

Embryology Boot Camp

The Embryology course was really amazing. It was an intensive 6 wk intro to animal development; we covered worms, flies & other arthropods, four vertebrates (chick, mouse, Xenopus, and zebrafish), ascidians, sea urchins, several annelids, ctenophores (comb jellies), cnidarians, clams, other lophotrochozoans, and more clams. The format was 2-4 hours of lecture a day, interspersed with 6-12 hours of lab work, 6 days a week; each day usually stretched from 9:30am 'til 2am.

Drinking -- occasionally heavy drinking -- was part of the course (par for the course? ;), as was late night swimming and almost weekly parties.

The instructors were all excellent. Each lecture was typically given by a different professor, and the professors were among the best in their field. We usually had an hour-long Q&A session after each lecture, so we could really get into their research. And in general the lectures were coordinated with the labs, so immediately after hearing about e.g. chick development, we could go into the labs and work with chick eggs.

The lab part of the course was stunning: each group of teaching assistants brought reagents and technique handouts, and were available throughout the lab period to answer questions and help with techniques. I've never had so many expert biologists waiting hand and foot on me ;).

Zeiss brought several million dollars worth of microscopes to the course, too, so I could play with scopes that I simply don't have daily access to even here at Caltech. They did keep on breaking their own scopes by swapping out hardware, but that just means that I'm now somewhat expert at troubleshooting their microscopes!

The best part of the course was the other students. There were 25 students total, and I got to know the other 24 students really well. It was definitely a boot-camp atmosphere, with the attendant intimacy and group dynamics. Good stuff. I expect to be seeing 'em all again over the next few years.

It's hard to really explain how great this course was, but that's enough gushing for now ;). I'll post some pretty pictures from there over the next few weeks

Antibes and the Cote d'Azur

If you've been there, you understand. If you haven't, go. The Nice/Antibes/Cannes area is fantastic. We stayed at the Three Palms, which I highly recommend.

For-work & for-fun programming

I'm slowly catching up on stuff. There are some twill issues that need to be dealt with. I'm also working on updating the Cartwheel documentation to fix the problems that an installer has been reporting. And sloooowly but surely I'm picking up the pieces of my research from before the summer: PhD or bust!

p.s. Looks like advogato's domain name has expired. Couldn't figure out anyway to renew it.

posted by Titus Brown

Friday, August 05, 2005

Review of C++ unit testing frameworks Not Python related, but pretty interesting nevertheless... Via Len: Exploring the C++ Unit Testing Framework Jungle

posted by Grig Gheorghiu

Good introductory article on SQLObject From IBM developerWorks.

posted by Grig Gheorghiu

Friday, July 22, 2005

Fifth SoCal Piggies meeting

The SoCal Piggies had their fifth meeting at USC on July 21st at 7:00 PM. Six Piggies attended -- Daniel Arbuckle, Diane Trout, Steve Williams, Mark Kohler, Grig Gheorghiu and Brian Leair.

Grig presented an overview of the py library, a collection of modules that intend to address several issues with the Python standard library. The py lib's mantra is "No API", which means it aims to be as simple to use and as "pythonic" as possible, while avoiding the F word ("Framework"). Grig talked mostly about py.test, and then briefly presented greenlets, py.xml and py.log. Hopefully he managed to convince the attendants to give the py lib a try.

We obviously also talked about Django, the hottest new thing since sliced bread in Python land. Daniel said he gave it a try, but he found the installation much harder than CherryPy's, so for now he's sticking with the latter. We went around the room in search of topics for future presentations, and fortunately it looks like there's no shortage of subjects: wxWindows, PIL, matplotlib, decorators, sets, and the list goes on.

We also briefly toyed with and solved the first 2 challenges of the Python Challenge, just to see how it feels to code together as a group. It was fun, so maybe we'll dedicate a future meeting to a coding session, be it the Python Challenge or other problems that can be solved in max. 1 hour.

Many thanks to Daniel for hosting the meeting and to Diane for bringing the projector.

The next meeting will be at USC again, with the date tentatively set to August 18th. Topics for next meeting:

posted by Grig Gheorghiu

Wednesday, June 29, 2005

PEP 238 - import .relative Python Imports

PEP 238 which has been accepted and may have an impact on any Python packages you've made so far, depending on your import style. This PEP will change the way imports work in a package. By default, all imports will be concidered absolute. Previously, if you had a Python package named 'Foo' with modules 'Bar' and 'Fig', you could import 'Fig' from the 'Bar' module by typing 'import Fig'. Once this new feature is implemented in Python 2.5, it will look for 'Fig' in sys.path first. They also plan to implement relative imports as well. With this you will be able to import the 'Fig' module from 'Bar' by typing 'import .Fig' for more a better example, see Guido's Decision or read the entire PEP. Other features include importing groups like: 

   1 from os import (path,
   2               sys,
   3               mkdir)

Which allows for easier multi-line imports. Check out the PEP for additional information.

Note that you can use the new features in Python 2.4 using:

In Python 2.5, any package that results in a relative import from a module from within a package will result in a DeprecationWarning. Python 2.6 will include the fully implemented and on by default PEP 238.

PEP 238 - Abstract

The import statement has two problems:

For the first problem, it is proposed that parentheses be permitted to enclose multiple names, thus allowing Python's standard mechanisms for multi-line values to apply. For the second problem, it is proposed that all import statements be absolute by default (searching sys.path only) with special syntax (leading dots) for accessing package-relative imports.

Feedback

Let me know if this post was useful or not, so I'll know wether or not to post something like this again. I hope everyone finds this post to be useful.

posted by Brandon King at 12:22 PM

Comments

PJE said...

You should've checked Python 2.4; from future import absolute_import doesn't exist.

The only part of PEP 238 implemented in Python 2.4 was the parentheses; everything else is deferred pending implementation. Also, I'm not 100% positive, but I think the fact that absolute imports aren't backward compatible really kills forced absolute importing prior to 3.0. (I personally will argue against it being implemented on that ground, as the current relative/absolute confusion is actually useful and used by Python programs now, e.g. to forcibly trick a module into seeing different module versions than it otherwise would.)

Grig Gheorghiu said...

Brandon,

I think your post is right on target as far as the goal as this blog is concerned. Keep posting :-) !!!

Daniel Arbuckle said...'

It may be that the future import will be implemented in upcoming versions of 2.4.

Brandon King said...

You'd think you could trust that an 'accepted' PEP would contain real and valid information. 

>>> from __future__ import absolute_import
File "<stdin>", line 1
SyntaxError: future feature absolute_import is not defined

Well, good thing someone checked that. Any way, I hope the rest of the information was valuable. I'll test 'implemented features' before posting next time.

Saturday, June 25, 2005

Fourth SoCal Piggies meeting

The SoCal Piggies had their fourth meeting at USC on June 21st at 7:00 PM. Seven Piggies attended -- in order of appearance: Daniel Arbuckle, Howard Golden, Grig Gheorghiu, Diane Trout, Mark Kohler, Charlie Hornberger and George Bullis.

Diane presented a tutorial on SimpleTAL, a Python-based templating language derived from Zope's ZPT. SimpleTAL allows you to lay out dynamic content in HTML or XML form. Diane showed us examples of XHTML templates that contained SimpleTAL-specific expressions such as tal:content, tal:condition, tal:repeat. The nice thing about SimpleTAL templates is that they can be edited by Web designers using their HTML editing/authoring tool of choice, and the tal expressions will simply be ignored by the editor. This enables a clean separation between presentation and content and lets programmers concentrate on the business logic aspect of the application (written in Python of course :-). Diane also showed us a demo of a Quixote-based Web application called CompClust that she developed at Caltech. CompClust is a bio-informatics app that uses various clustering algorithms for studying relationships between genes. The app also makes heavy use of matplotlib to generate some very nifty graphics on the fly. The templating engine for the application is of course SimpleTAL.

The second speaker of the night was...Diane again. She presented a tutorial on Pyrex, a language for writing Python extension modules. Pyrex allows you to write code that mixes Python and C data types, and compiles it into C code as a Python extension module. Diane used Pyrex in her bio-informatics Web application to call C code that runs clustering algorithms from her Python code. One less known use of Pyrex is to compile Python programs into binaries that can then be used as stand-alone executables without the Python interpreter. George asked what does Pyrex do that SWIG doesn't, and Diane said Pyrex is more forgiving when it comes to interfacing with C code, and also allows you to more easily write Python code that interacts with the C function calls. For more details on Pyrex, see also this Charming Python tutorial from IBM developerWorks.

[Note on viewing the S5-based slides that Diane put together: for the controls that allow you to move back and forth through the slides, move the mouse cursor to the middle bottom part of the screen]

We also had some lively discussions on ways to dynamically generate HTML and possibly reuse HTML snippets across an application, especially in the case when a template mechanism would be too static (a subject brought up by Mark). Several possibilities were mentioned: the htmlgen module, good ol' CGI, rolling your own acquisition mechanism similar to the one used in Zope, and of course the more heavy-weight approach of using a Web application such as Quixote. The subject of GUI widget toolkits was also brought up, especially ways to build GUIs in a cross-platform, cross-tool way by specifying for example the names and values of the widgets in an XML file. One could then ideally use any GUI toolkit in order to actually display the widgets on the screen. XUL was mentioned as one technology to look at in this arena.

We also discussed a bit about unit testing, and I mentioned py.test as an alternative to the standard unittest module. I think a presentation on Python's unit test frameworks would be interesting, I wrote some tutorials that I can put together as a slide show for a future meeting.

Many thanks to Diane for her presentations and to Daniel for hosting the meeting.

The next meeting will be at USC again, on a date to be announced. Topics for next meeting:

Posted by Grig Gheorghiu

Sunday, June 12, 2005

twill release (Repost, via the socal-piggies blogger stuff:)

twill 0.7.1

Coooooome and get it! Announcement, docs, download.

This is basically a bug-fix version with about 15 different minor fixes. Several users requested things like HTTP basic auth handling and file uploads, and I patched several bugs in the underlying packages (urllib etc.) Internal ugliness was fixed or augmented, depending on whether or not I was dealing with forms, which is inherently ugly code.

Once I've got all the desired features in there, I can start cleaning up the internal code; right now I'm focused more on testing what's there (which will make cleanup easier!) I've also gotta figure out a way to do meaningful unit tests. That's for v0.8.

For the summer, I will be gone. I predict I will not be working on twill until I get back in late August...

posted by Titus Brown

Saturday, June 11, 2005

Simple function optimizaton

This is an optimization technique I learned about a few days ago at a symposium. "Optimization" in this case means finding the input parameter values that result in the best function output, or at least approximating them.

The performance of this implementation isn't as good as the presenters', in terms of the number of iterations it takes to approach the optimum. I'm not sure what the cause of that is, but it may well just be a case of their having optimized their optimizer (two different meanings for the word "optimize" in the same sentence... there oughta be a law.)

In any case, this is a pretty nice algorithm, which converges relatively fast and (perhaps more importantly) is quite simple and understandable. Once you understand it, you'll be able to code it out from memory.

The general idea is that the optimizer maintains a "swarm" of "particles" which move about the space of the function, trying to find the best locations. The rules used to control their movements are (loosely) inspired by those that guide the flocking behavior of birds and fish.

This implementation is perhaps obfuscated by the use of numarray, but the advantages are too great to ignore. 

   1 from numarray import array, greater, maximum, minimum, transpose
   2 from numarray.random_array.RandomArray2 import random as randarray
   3 
   4 def pso(function, ranges, particle_count = 5, friction = 0.95, confine = True):
   5   r"""
   6   Performs a Particle Swarm Optimization on the passed function,
   7   performing one optimization step for each iteration step. The
   8   yielded values are are 2-tuples of (best known value, tuple of
   9   parameters resulting in the best known value).
  10 
  11   Parameters:
  12 
  13   * function: The function you wish to find the optimal inputs
  14     for. It should be a function of X floating point parameters,
  15     where X = len(ranges). Further, the result should be a floating
  16     point number which can be interpreted as the function quality
  17     with a given set of parameters. If your desired function does
  18     not result in a float or is not parameterized by floats, feel
  19     free to wrap it in a function that does appropriate translation.
  20 
  21   * ranges: The bounds within which the exploratory particles will
  22     be created. If confine = True, these ranges also bound the space
  23     to be explored. Each range should be a 2-tuple of (lower bound,
  24     upper bound), and there should be one for each parameter of the
  25     function.
  26 
  27   * particle_count: The number of particles to maintain while
  28     optimizing. Fairly small numbers are often good. Try 5 if you're
  29     not sure.
  30 
  31   * friction: The amount of 'velocity' that is lost from the system
  32     on each time step. Friction will be clamped to the range [0-1],
  33     and then mutuplied by the current particle velocities. A value
  34     of 1 will tend to cause the optimizer to fail if the boundary
  35     flag is not set to optimize.reflect. If you're not sure what value to
  36     use, try 0.95
  37 
  38   * confine: May be set to False, allowing exploration beyond the
  39     bounds set by the ranges parameter.
  40   """
  41 
  42   dimensions = len(ranges)
  43   upper = array([r[1] for r in ranges])
  44   lower = array([r[0] for r in ranges])
  45   shape = (particle_count, dimensions)
  46   positions = randarray(shape)
  47   for pos, rng in zip(positions, ranges):
  48       pos *= rng[1] - rng[0]
  49       pos += rng[0]
  50   velocities = randarray(shape) / randarray(shape)
  51   lbest = array([function(*pos) for pos in positions])
  52   lbestat = array(positions)
  53 
  54   gbestat = lbest.argmax()
  55   gbest = lbest[gbestat]
  56   gbestat = array(positions[gbestat])
  57 
  58   while True:
  59       velocities *= friction
  60       velocities += randarray(shape) * (array([gbestat]).repeat(particle_count) - positions)
  61       velocities += randarray(shape) * (lbestat - positions)
  62 
  63       positions += velocities
  64 
  65       if confine:
  66           positions = minimum(maximum(positions, lower), upper)
  67 
  68       values = array([function(*pos) for pos in positions])
  69       replace = greater(values, lbest)
  70       lbest[replace] = values[replace]
  71       lbestat[replace] = positions[replace]
  72 
  73       mval = values.max()
  74 
  75       if mval > gbest:
  76           gbest = mval
  77           gbestat = positions[values.argmax()]
  78 
  79       yield gbest, gbestat

posted by Daniel Arbuckle

Comments

Daniel Arbuckle said...

Whoops. Somewhere before the while loop, the following line should appear: 

   1 friction = min(max(friction, 0.0), 1.0)

Friday, May 13, 2005

Cool *nix based backup tool - written in Python

rdiff-backup - A remote incremental backup of all your files could be as easy as

"rdiff-backup / host.net::/target-dir"

A friend turned me on to this tool, seems to have nice features compression, runs over ssh, you can backup between 2 remote machines, incremental back etc... A list of its features can be found here:

rdiff-backup features

Looks like someone tried to use it on windows in cygwin to a limited effect. rdiff-backup seems to work as billed on linux.

posted by neener

Friday, May 06, 2005

Inserting data into the parent process

On Linux, and other UNIX-like systems that support it, you can make an ioctl call to insert data into your standard input file. That's pretty neat in an of itself, but it gets better: in most cases, you share the standard input of your parent process. What that means is that you can insert data into that file, and then terminate... and the parent process will read those data as user input.

The following Python snippet will, when run from a command shell, cause that shell to print a file listing of the current directory. 

   1 #! /usr/bin/env python
   2 from fcntl import ioctl
   3 
   4 TIOCSTI = 0x5412
   5 
   6 ioctl(1, TIOCSTI, 'l')
   7 ioctl(1, TIOCSTI, 's')
   8 ioctl(1, TIOCSTI, '\n')

This trick has security implications, and is one reason why programs that run as root should never pass their open file handles on to their children.

posted by Daniel Arbuckle at 7:50 AM

Comments

Blogger Grig Gheorghiu said...

Very neat. I'm curious in what context you used this trick.

Blogger Daniel Arbuckle said...

I was writing a chording driver for an n52 gamepad, to make it usable as a full keyboard.

I didn't actually go with this method, but I thought y'all might get a kick out of it.

Wednesday, May 04, 2005

Manipulating Windows registry values

Here is a small class that connects to the Windows registry and creates sub-keys and/or values for a given registry key. I used this functionality very recently to add a directory to the PATH environment variable by modifying the Environment\Path registry value.

Update 06/08/05: When modifying the 'Path' registry value, we need to set a value type of REG_EXPAND_SZ and not REG_SZ, otherwise variables like %SystemRoot% will not get correctly expanded by the command prompt interpreter and the WINDOWS\system32 directory will not be found. I added a value_type parameter defaulting to REG_SZ to the set_value method. 

   1 import _winreg
   2 class WinRegistry:
   3 
   4   def __init__(self, reg=_winreg.HKEY_LOCAL_MACHINE,
   5                key=r"System\CurrentControlSet\Control\Session Manager\Environment"):
   6       self.reg = _winreg.ConnectRegistry(None, reg)
   7       self.key = key
   8 
   9   def create_subkey(self, subkey):
  10       key = self.key + "\\" + subkey
  11       _winreg.CreateKey(self.reg, key)
  12 
  13   def delete_subkey(self, subkey):
  14       key = self.key + "\\" + subkey
  15       _winreg.DeleteKey(self.reg, key)
  16 
  17   def set_value(self, value_name="", value="", subkey=None, value_type=_winreg.REG_SZ):
  18       key = self.key
  19       if subkey:
  20           key += "\\" + subkey
  21       k = _winreg.OpenKey(self.reg, key, 0, _winreg.KEY_WRITE)
  22       _winreg.SetValueEx(k, value_name, 0, value_type, value)
  23       _winreg.CloseKey(k)
  24 
  25   def get_value(self, value_name="", subkey=None):
  26       key = self.key
  27       if subkey:
  28           key += "\\" + subkey
  29       k = _winreg.OpenKey(self.reg, key)
  30       value = _winreg.QueryValueEx(k, value_name)[0]
  31       _winreg.CloseKey(k)
  32       return value
  33 
  34   def delete_value(self, value_name="", subkey=None):
  35       key = self.key
  36       if subkey:
  37           key += "\\" + subkey
  38       k = _winreg.OpenKey(self.reg, key, 0, _winreg.KEY_WRITE)
  39       _winreg.DeleteValue(k, value_name)
  40       _winreg.CloseKey(k)
  41 
  42 if __name__ == '__main__':
  43   wr = WinRegistry()
  44 
  45   # Add ApplicationPath1 value_name to the Environment key
  46   value_name="ApplicationPath1"
  47   value="C:\\agent1\\agent.bat"
  48   wr.set_value(value_name, value, value_type=_winreg.REG_EXPAND_SZ)
  49   value = wr.get_value(value_name)
  50   print "Created value_name %s with value %s" % (value_name, value)
  51   # Now delete value_name
  52   print "Now deleting", value_name
  53   wr.delete_value(value_name)
  54 
  55   # Create subkey Environment\EnvTest1 and add ApplicationPath2 value_name to it
  56   subkey = "EnvTest1"
  57   value_name="ApplicationPath2"
  58   value="C:\\agent2\\agent.bat"
  59   wr.create_subkey(subkey)
  60   print "Created subkey", subkey
  61   wr.set_value(value_name, value, subkey)
  62   value = wr.get_value(value_name, subkey)
  63   print "Created value_name %s with value %s" % (value_name, value)
  64   # Now delete subkey
  65   print "Now deleting", subkey
  66   wr.delete_subkey(subkey)

posted by Grig Gheorghiu

Saturday, April 30, 2005

darcs rocks

I've been experimenting with darcs since Titus mentioned it at the last SoCal Piggies meeting, and my conclusion is that it's the best revision control tool I've seen, for personal-scale projects. I haven't really stress-tested it for project size scalability yet, so that's where the possible caveat lies. I've heard that it couldn't handle the Linux kernel... but then, not many things can.

If you don't already use a revision control tool, or if you're not quite happy with the one you've got, give darcs a shot.

Posted by Daniel Arbuckle

Blogger Grig Gheorghiu said...

Could you post a 5-minute guide of installing darcs, creating a repository and checking out/checking in files?

Blogger Daniel Arbuckle said...

Installing depends on your platform. On my Gentoo Linux box, it was a matter of typing 'emerge darcs' and waiting for a while.

On Windows, I downloaded a binary, plus followed a few simple instructions to enable ssh for repository access.

There are good instructions for downloading and installing on many systems.

Once you have darcs installed, you'll want to set up a repository. You do that by going to the root of the respository-to-be's directory tree and typing 'darcs initialize'

Then you 'darcs add' any files that you want to be version controlled

Once all the files are in place and added, execute 'darcs record' and you're setting up

To get a local copy of a darcs repository, run 'darcs get LOCATION' where LOCATION is a URL, a filesystem path, or an ssh remote path

After you've changed some things, you run 'darcs record' again to store those changes in revision control.

Finally, to update repository A with the changes made to repository B, you can use 'darcs push A_LOCATION' from within B, or 'darcs pull B_LOCATION' from within A

Blogger Daniel Arbuckle said...

The workflow with darcs follows this pattern when you're working in a group:

When you're working alone, you just create a repository and then record your changes whenever you feel that it's appropriate.

Blogger Daniel Arbuckle said...

Oh yeah, when you're working with others, its sometimes useful to pull their changes into your repository, too :)

Thursday, April 28, 2005

Poor man's Web services with HTTP POST

I use HTTP POST as a lightweight mechanism for posting test results to a central location. I have tests running on various clients/platforms/OSes, and I need a way to keep track of the results of those tests. For this, I have an Apache server running CGI scripts that talk to a Firebird database. By using HTTP POST from the clients to the Apache server, I keep things simple, because the clients don't need to have any database bindings or drivers installed. In fact, all the clients need to know is how to open a socket on port 80 to the Web server.

This can be looked at as a poor man's Web service mechanism.

Here is the http_post function I'm using: 

   1 def http_post(data, cgi_cmd, webserver=""):
   2      if not webserver:
   3          webserver = "mywebserver.mydomain.com"
   4      if type(data) == type(u''): # unicode
   5          data = data.encode('iso-8859-1')
   6 
   7      sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
   8      try:
   9          sock.connect((webserver, 80))
  10      except:
  11          print "Error: Could not connect to " + webserver
  12          return
  13 
  14      url = "http://%s/cgi-bin/%s" % (webserver, cgi_cmd)
  15 
  16      sock.send("POST %s HTTP/1.0\n" % url)
  17      sock.send("Accept: */*\n")
  18      sock.send("User-Agent: http_post_mechanism\n")
  19      sock.send("Content-Type: application/x-www-form-urlencoded\n")
  20      sock.send("Content-Length: %d\n\n" % len(data))
  21      sock.send(data)
  22      while 1:
  23          response = sock.recv(8192)
  24          if not response: break
  25          output = response
  26      sock.close()

posted by Grig Gheorghiu at

Wednesday, April 27, 2005

Hello Piggies I thought it would be a good idea to get a blog for the SoCal Piggies (Southern California Python Interest Group). This will be a "team blog", where all us Piggies will be able to post. I hope we will all contribute by posting for example short "Python tips and tricks" that can make our lives easier as Python users.

Posted by Grig Gheorghiu

BlogHistory (last edited 2010-10-05 15:18:53 by iexpert-67)