Why privacy matters and State Secrecy not

Today i saw a woman’s german article on how she wouldn’t want reports about her sex life spread through wikileaks. I wouldn’t like that either. But i see it as a misconception and confusion about two very different issues, namely personal privacy and state/government secrecy. This confusion is used and increased by many politicians for their own purposes.

Privacy means our rights to have a private life, private flat, private actions and private communications. We don’t want the state or the public to surveil us or intrude our private world unless they can show evidence to an independent court that there is something criminal going on.

State secrecy denotes making secret deals, performing secret communication with and secret actions against people or other states. Secret actions can naturally not be discussed in the public and are exempt from our judgement when electing officials. Increasing state secrecy very quickly leads to inner circles wielding great power. The 20th century has tons of bad examples.

Therefore I refuse the notion that if am positive about privacy i must also be ok with state secrecy. Or if i want a transparent government that i also must be ok with total surveillance of my private life. No way. Likely it’s rather true that the more transparent a government is the more secure i can feel with respect to my privacy.

On a sidenote, this all relates to a point in the hacker ethics made by Wau Holland and the Chaos Computer Club a long time ago: “Make public data available, protect private data.”

Wikileaks or: Welcome to the brave new “Terms of Use” era

Paypal “restricts” the bank account of the german Wau Holland foundation which managed parts of Wikileaks transactions. Citing from their statement :

PayPal has permanently restricted the account used by WikiLeaks due to
a violation of the PayPal Acceptable Use Policy, which states that
our payment service cannot be used for any activities that
encourage, promote, facilitate or instruct others to engage in
illegal activity. We’ve notified the account holder of this action.

As with my yesterday notes a number of questions arise. What exactly is
illegal about wikileaks behaviour in the US? Was money laundering or
online fraud involved? Was wikileaks given a warning and asked to
provide clarifying statements regarding the usage of _their_ money? Do
they apply the same moral standards to all their customers and accounts?
For example, are they also closing all accounts related to the
Washington Times or related to Sarah Palin publically calling for
assasination of a citizen of Australia?

If we are learning anything from the Wikileaks case it is the willfullness
of young IT companies in the US to concur with public opinion or direct
government pressure. Up until now i presumed the likes of Amazon or Paypal
also incorporated some ethics. And this would mandate IMO to keep
distance to government actions and to resist government pressure especially
if it’s about something like wikileaks. It seems like the US goverment
used its advanced notice of the pending publication wisely: rather than filtering critical
messages they organised a huge campaign in the US resulting in Amazon,
PayPal and everydns and probably others to implement government policies
and quickly. No need to have special laws or evil government actions – just reference the Terms of Use and be done.

Who needs censorship if you have Amazon and everydns?

Yesterday Amazon stopped services for the wikileaks archive. Citing from the Amazon statement:

It is not credible that the extraordinary volume of 250,000 classified documents that WikiLeaks is publishing could have been carefully redacted in such a way as to ensure that they weren’t putting innocent people in jeopardy.

So here you have a popular cloud services provider judging their customers content in a broad manner, stating conclusion, terminate services, done. Did they allow discussion? Do journalists need to keep and control all rights of the material they are publishing? Can Amazon show any evidenceof their “jeopardy” conclusion? Would they apply the same moral standard to e.g. blogs or other (journalistic) content that called for going to the war on Iraq – causing >100.000 civilians to die? Does Amazon really want us to believe the US governments outrage and Mr. Liebermanns actions are merely a co-incidence?

Then today everydns.net terminated DNS services for wikileaks.org with a 24 hour prior notice. Citing from the everydns statement:

More specifically, the services were terminated for violation of the provision which states that “Member shall not interfere with another Member’s use and enjoyment of the Service or another entity’s use and enjoyment of similar services.”

So supposedly there was an attack on their DNS servers and they interpreted this as wikileaks doing harm to others? Is this the future of how we handle attacks against single domain names? Would the same happen if it was the site of a chinese dissident or a Poker playing site or any other customer?

As things stand this puts me off these two companies and brings me again to the thought that we need an internet that is as independent from any single company or any single country as possible - it’s in the best interest for all of us in the long run.

Ring of Python talk at pycon

Just did my talk on Ring of Python talk at Pycon US 2010, discussing competition and features of Python interpreters and co-operation issues around the most important issue in my oppinion: deployment. Also showcased execnet as a generic Python2Python bridge, connecting Python2.4, Python 3.1, Jython and IronPython. Got some nice feedback, also about the presentation style, actually i was using Prezi. You may go to the following page, click into the flash app and hit “cursor right”: Ring of Python .

Elastic Python deployment networks

Time for a bit of fiction on distributed Python deployment. As some of you know, py.execnet imperatively and elastically executes code in local or remote python processes, maintaining channels for exchanging data. Execnet has the wonderful zero-install feature which means no software except a Python interpreter is required remotely. The connection between Python interpreters is direct, i.e. the connecting side needs to know how to start the remote side. And it’s non-transitive meaning: given a A->B and a B->C connection there is no support for getting a A->C connection mediated by B.

I’d like to lift these restriction and introduce the concept of a deployment network through which execnet connections can be mediated. I am pondering a command line tool that creates a network of Python intepreters on multiple hosts like this:

execnet start mynet ssh=linuxbox.org socket=windowsbox.com ssh=osx.com

This would create a "mynet" deployment network of four Python interpreters running on different hosts and platforms: one local process and three remote processes connected to it. Let’s add a remote Jython process to the "mynet" deployment network:

execnet addhost mynet ssh=remote//python=jython

We can generally use the ‘mynet’ handle to work with this newly instantiated deployment network. For example, to get a fresh process on a certain platform from a Python program:

mynet = execnet.connect('mynet')
gateway = mynet.makegateway(platform="java")

The first line connects to our local ‘mynet’ process. The second line creates a gateway to a fresh Python interpreter, in this case a Jython process. The bootstrapping of the Jython-side gateway object is determined by the initiating client side. The two subprocesses communicate through an IO-connection that is mediated by the ‘mynet’ deployment network.

This is very exciting because the zero-installation feature is preserved on two levels: the deployment processes work on software coming from a single point, the command line above. And our "on-top" gateways operate with software determined from the initiating side, from the python code above. Interaction between the two worlds is limited to a connect operation and providing IO mediatiation. This means the deployment network facilities can evolve independently from the "on-top" execnet-elastic programs.

Conceptually it’s a very reliable and robust setting. The mynet processes should be able to run as robustly as unix shells. They are to provide a solid base for writing and deploying Python applications that span multiple interpreters. They don’t run applications in-process.

This is not all fiction. The current development version of py.execnet already works across and between CPython2.4 through to CPython 3.1, Jython and PyPy. And i intend to release execnet as a separate package soon, providing the basis for implementing the above fiction and lots of other on-top fun

Let’s prevent a no-privacy world by better technology!

Dear Google, Amazon and web 2.0: you are doing a great job of providing cool services and apps, you are doing away with obnoxious installation and upgrade steps, your stuff often works out of the box and you offer convenient interfaces. Great.

Comes with a caveat, though: the loss of control of my data and my communication, loss of ability to install and run whichever programs I like. You as central organisations and your ruling governments develop the practical possibility to get at all this data, on a mass basis and retro-actively. Something needs to change about this or we all will be ending in a world with historically unprecedented power structures in the hand of few. A world with virtually no privacy.

Part of what i can do is thinking about cool new technology to counter these developments. I am convinced we need a more decentral application execution infrastructure. We need open cloud software and infrastructure that allows to have easy-to-develop apps run "on the net" – a wealthy network of PCs and mobile phones. A free wireless-type network, not expensive mobile "total control" networks.

pylib 1.0.0 released: the testing-with-python innovations continue

Took a few betas but finally i uploaded a 1.0.0 py lib release, featuring the mature and powerful py.test tool and "execnet-style" elastic distributed programming. With the new release, there are many new advanced automated testing features – here is a quick summary:

• funcargs – pythonic zero-boilerplate fixtures for Python test functions :
  • totally separates test code, test configuration and test setup
  • ideal for integration and functional tests
  • allows for flexible and natural test parametrization schemes
• new plugin architecture, allowing easy-to-write project-specific and cross-project single-file plugins. The most notable new external plugin is oejskit which naturally enables running and reporting of javascript-unittests in real-life browsers.
• many new features done in easy-to-improve default plugins, highlights:
  • xfail: mark tests as "expected to fail" and report separately.
  • pastebin: automatically send tracebacks to pocoo paste service
  • capture: flexibly capture stdout/stderr of subprocesses, per-test …
  • monkeypatch: safely monkeypatch modules/classes from within tests
  • unittest: run and integrate traditional unittest.py tests
  • figleaf: generate html coverage reports with the figleaf module
  • resultlog: generate buildbot-friendly reporting output
  • …
• distributed testing and elastic distributed execution:
  • new unified "TX" URL scheme for specifying remote processes
  • new distribution modes "–dist=each" and "–dist=load"
  • new sync/async ways to handle 1:N communication
  • improved documentation

The py lib continues to offer most of the functionality used by the testing tool in independent namespaces.

Some non-test related code, notably greenlets/co-routines and api-generation now live as their own projects which simplifies the installation procedure because no C-Extensions are required anymore.

The whole package should work well with Linux, Win32 and OSX, on Python 2.3, 2.4, 2.5 and 2.6. (Expect Python3 compatibility soon!)

For more info, see the py.test and py lib documentation:

http://pytest.org

http://pylib.org

have fun, holger

code-centered issue tracking?

Is there anything that allows code-centered issue tracking? Recently, Gustavo Niemeyer had an interesting piece up where he suggests private/protected syntax for Python. His point is that code collaboration doesn’t otherwise work in larger uncontrolled environments of dev groups. I agree with most of his observations but not his conclusion. I’d rather like to see reduced communication costs for changing code. Here is an example of what i mean. If see a code fragment like this:

def somefunc(self, x, y, z):
    self._cache = func(x) + other(y)
    self.z = self._cache + third(z)

i want to be able to visually mark this code, write a comment like:

hey, i need the third(z) value, and doing self.z-self._cache feels bad – can you help?

and i want my development environment to automatically route this question with exact code refs to the maintainers of the code. This should not take longer than 20 seconds and be automatically managed.

On the receiving side, as the maintainer, i want to get notified and be able to say:

getissues mypkg/subpkg

and have it automatically list me all files and issues for it. So i easily see the above question, do a patch, and issue:

sendpatch PATCHNAME

and type in a message. Maybe automatically CCed to other library maintainers, a mailing list etc. As the original sender i get back a mail and can use a cmdline tool to apply the patch, give feedback and so forth …

IOW, i want to have tools that automatically manage the issue-addressing, code-referencing, finding-out about package info/release numbers, format the comment related to the cited code, send the mail, register an issue automatically and so on. Is there something like this?

If not: doesn’t we already have most of the pieces? What makes sense to use for it? Mercurial and Patch queues? Maybe the new Bitbucket API? Integrate with existing issue tracker? First goal would be to have it manage itself, i guess

cheers, holger

Putting test-hooks into local and global plugins

(updated to match 1.0 API and features) I’d like to clarify py.test’s hook lookup and showcase the ease of writing things into per-project or global plugins. The pytest_generate_tests was discussed in the last blog post on parametrizing tests. Here is where you can write this hook down:

• in the test module
• in a local plugin
• or in a global plugin

The last blog post showed how to put the hook directly into the test module. Let’s take a look now what putting a hook into a "local or global plugin" means.

Putting a Hook in a local plugin

Putting the generate-hook into a local plugin means to create a conftest.py file in your test directory or package root directory with these contents:

#./conftest.py
def pytest_generate_tests(metafunc):
     # exactly same implementation as module-level one

As with test files and functions and arguments, conftest.py files and the exact name ConftestPlugin will be automatically discovered and the plugin be instantiated.

Put hook into a global plugin

Putting the generate-hook in a global cross-project plugin requires to invent a file or package name with a fixed pytest_ prefix. Here is how you would write down the generate hook into a self-contained pytest_mygen.py file:

#./pytest_mygen.py

def pytest_generate_tests(metafunc):
      # exactly same implementation as module-level one

The hook name including its metafunc argument needs to be used exactly as decribed – loading your plugin will otherwise result in an error.

Activating a global plugin

While local plugins are automatically discovered, global plugins need to specified. To activate a global plugin pytest_mygen you can use either of the following three ways:

py.test -p mygen             # for command line activation

export PYTEST_PLUGINS=mygen  # for shell/env activation

pytest_plugins = "mygen"     # in a test module or conftest.py

py.test loads command-line or environment-specified plugins very early so that plugins can add command line options.

multiple pytest_generate_hook implementations

All existing pytests_generate_hooks hooks will be called once for each test function. You can have multiple hooks but the generate-hook usually only acts on a specific funcarg by doing a check like this:

if "myarg" in metafunc.funcargnames:
    ...

So, you say, what about a test function with multiple arguments – could each function argument come from a different generating provider factory?

This would mean that multiple generators act independently but want to collaborate and combine their values for a given test functions. Well, if you encounter a real need for it, please come forward and we’ll think up a fitting API extension. A couple of days ago i had a "combining funcargs" API call implemented but decided to remove it because i try hard these days to only add features that have a proven need.

Putting test support code into plugins FTW

Lastly, let me point out that putting the pytest_generate_tests hook into a plugin allows the actual test code to stay igorant about the exact way how or where parametrization is implemented. For example, adding command line options for influencing the generation or selection of parameter sets, including randomizing, would not change a single character in the test module and test code.

have fun and let me know what you think, holger

Parametrizing Python tests, generalized.

Parametrizing test runs is a kind of a hot topic with Python test tools. py.test recently grew a new pytest_generate_tests hook to parametrize tests. I am going to introduce it by providing ports of Michael Foord‘s recent experiments with parametrizing unittest.py test cases and an example from Rob Collins testscenarios unittest extension. The gist of the new hook is that it allows to easily implement and combine these schemes. It builds on the general idea of allowing python test functions to receive function arguments ("funcargs") – and defining mechanisms on how to provide them.

The parametrizer example, ported

The idea of Michael Foord‘s Parametrizer example is to define multiple sets of parameters and have specified test functions receive those arguments. Here is a direct port of Michael’s example to use py.test’s new hook:

#./test_parametrize.py
import py

def pytest_generate_tests(metafunc):
    # called once per each test function
    for funcargs in metafunc.cls.params[metafunc.function.__name__]:
        # schedule a new test function run with applied **funcargs
        metafunc.addcall(funcargs=funcargs)

class TestClass:
    params = {
        'test_equals': [dict(a=1, b=2), dict(a=3, b=3), dict(a=5, b=4)],
        'test_zerodivision': [dict(a=1, b=0), dict(a=3, b=2)],
    }

    def test_equals(self, a, b):
        assert a == b

    def test_zerodivision(self, a, b):
        py.test.raises(ZeroDivisionError, "a/b")

py.test automatically discovers both the pytest_generate_tests hook and the two test functions. For each test function it calls the hook, passing it a metafunc object which provides meta information about the test function and allows to add new tests during collection. Let’s see what just collecting the tests produces:

$ py.test --collectonly test_parametrize.py

<Module 'test_parametrize.py'>
  <Class 'TestClass'>
    <Instance '()'>
      <FunctionCollector 'test_equals'>
        <Function 'test_equals[0]'>
        <Function 'test_equals[1]'>
        <Function 'test_equals[2]'>
      <FunctionCollector 'test_zerodivision'>
        <Function 'test_zerodivision[0]'>
        <Function 'test_zerodivision[1]'>

So we collected 5 actual runs of test functions. Let now run the test functions:

$ py.test test_parametrize.py

========================= test session starts =========================
python: platform linux2 -- Python 2.6.2
test object 1: test_parametrize.py

test_parametrize.py F.F.F

============================== FAILURES ===============================
________________ TestClass.test_equals.test_equals[0] _________________

self = <test_parametrize.TestClass instance at 0x994f8ac>, a = 1, b = 2

    def test_equals(self, a, b):
>       assert a == b
E       assert 1 == 2

test_parametrize.py:14: AssertionError
________________ TestClass.test_equals.test_equals[2] _________________

self = <test_parametrize.TestClass instance at 0x994f8ac>, a = 5, b = 4

    def test_equals(self, a, b):
>       assert a == b
E       assert 5 == 4

test_parametrize.py:14: AssertionError
__________ TestClass.test_zerodivision.test_zerodivision[1] ___________

self = <test_parametrize.TestClass instance at 0x994f8ac>, a = 3, b = 2

    def test_zerodivision(self, a, b):
>       py.test.raises(ZeroDivisionError, "a/b")
E       ExceptionFailure: 'DID NOT RAISE'

test_parametrize.py:17: ExceptionFailure
================= 3 failed, 2 passed in 0.13 seconds =================

You can easily see the failing tests and the parameters that the tests received. It also showcases py.test traceback reporting but that’s for another discussion.

The parametrizer example, decorated

So, you say, what about having a decorator specifying test parameters? Here is the same example, letting our hook implement a decorator scheme:

#./test_parametrize2.py

import py

def params(funcarglist):
    def wrapper(function):
        function.funcarglist = funcarglist
        return function
    return wrapper

def pytest_generate_tests(metafunc):
    for funcargs in getattr(metafunc.function, 'funcarglist', ()):
        metafunc.addcall(funcargs=funcargs)

# actual test code, above support code can live elsewhere

class TestClass:
    @params([dict(a=1, b=2), dict(a=3, b=3), dict(a=5, b=4)], )
    def test_equals(self, a, b):
        assert a == b

    @params([dict(a=1, b=0), dict(a=3, b=2)])
    def test_zerodivision(self, a, b):
        py.test.raises(ZeroDivisionError, "a/b")

This variant leaves the "test specification" tightly coupled. Running it with py.test test_parametrize2.py provides the some output as for the first example port.

A quick port of "testscenarios"

Finally, let’s also port Rob Collin’s testscenario example. Here is the implementation of the full mechanism with py.test and the tests in funcarg-style:

#./test_parametrize3.py

def pytest_generate_tests(metafunc):
    for scenario in metafunc.cls.scenarios:
        metafunc.addcall(id=scenario[0], funcargs=scenario[1])

scenario1 = ('basic', {'attribute': 'value'})
scenario2 = ('advanced', {'attribute': 'value2'})

class TestSampleWithScenarios:
    scenarios = [scenario1, scenario2]

    def test_demo(self, attribute):
        assert isinstance(attribute, str)

Let’s run it:

$ py.test -v test_parametrize3.py

================================ test session starts ================================
python: platform linux2 -- Python 2.6.2 -- /usr/bin/python
test object 1: test_parametrize3.py

test_parametrize3.py:14: TestSampleWithScenarios.test_demo[basic] PASS
test_parametrize3.py:14: TestSampleWithScenarios.test_demo[advanced] PASS

============================= 2 passed in 0.06 seconds ==============================

Easy, isn’t it?

Playing yourself

If you want to play with the examples yourself, you can use hg clone https://bitbucket.org/hpk42/py-trunk/ and setup.py install it. In the example/parametrize/ direcrory you can tweak and run the test examples. Let me know of comments or problems you may encounter.

Conclusion: deprecating "yield"

The three ports show that pytest_generate_tests is a hook that allows to implement many custom parametrization schemes. You can implement the hook in a test module or in local or global plugin, sharing it in your project or in the community. The hook also integrates well with other usages of funcargs, see the extensive pytest funcarg documentation.

The new way to parametrize test is meant to substitute yield usage of test-functions aka "generative tests", also used by nosetests. yield-style Generative tests have received criticism and despite being the one who invented them, i mostly agree and recommend not using them anymore.

I’d like to thank Samuele Pedroni and Ronny Pfannschmitt who helped to evolve the new hook and pushed me for implementing it. Oh, and did i emphasize that working feedback-based and documentation driven is so much better than going wild on hypothetical usages?

have fun, holger