Piotr Pokora

Vala bindings for Midgard2

Piotr Pokora — 2010-02-10T12:57:35+00:00

Today I got some help on #vala irc channel (thanks nemequ!), and I must say I am very happy to see Midgard API working with vala bindings. I am also happy cause in this case, there's no need to write bindings in C (as happens with PHP or Python bindings).

First of all, you have to have Midgard2 installed. Then, all you have to do, to have vapi file is to run simple script in vala directory . And finally, you can try example. Note, you must have database created. Midgard.Storage is excluded from vala bindings at this moment.

I hope to see mono bindings soon.

Mjolnir - new stable Midgard2 release

Piotr Pokora — 2009-11-18T13:02:12+00:00

Yes, we did it :)

Here is it latest and greatest Midgard2 stable release.

More flexibility for Midgard schemas

Piotr Pokora — 2009-10-08T12:36:30+00:00

Here it is. Well known Midgard MgdSchema is no longer such strict and limited. Since [Mjolnir][2] we will support user defined values which can extend class property even better. Till now, we supported only 'description' keyword, which allowed to add any text which could describe property.

Defining own values is plain and simple:

<type name "my_best_class" table="my_best_class_tbl">
    <property name="superproperty" type="string" field="super_property">
        <foo>bar</foo>
        <private>true</private>  
    </property>
</type>

Now, with API calls, one is able to get it quickly:

MidgardObjectClass *klass = g_type_class_peek (g_type_from_name ("my_best_class"));
MidgardReflectionProperty *mrp = midgard_reflection_property_new (mgd, klass);
midgard_reflection_property_get_user_value (mrp, "superproperty", "private");

Python:

mrp = midgard.reflection_property ("my_best_class")
print mrp.get_user_value ("superproperty", "private")

or even PHP:

$mrp = new midgard_reflection_property ("my_best_class");
echo $mrp->get_user_value ("superproperty", "private");

Note, that I used private key by purpose. If you are developing Midgard derived application and you need some functionality which can be defined in schema (but it's not yet), or work on such functionality is in progress, and you need to make your application be aware of this (quickly), you can depend on self added values.

Python bindings and GObject derived class destructor

Piotr Pokora — 2009-09-29T18:12:55+00:00

Tero found interesting memory issue with Midgard2 Python bindings. The code he presented is very simple:

import _midgard as midgard

while True:
    qb = midgard.query_builder ("midgard_host")

Guess, what happens. Memory usage grows up till your machine is frozen. I started to investigate this issue and reproduced problem with different case:

import gobject

class my_memory_test (gobject.GObject):
    def __init__ (self):
        gobject.Object.__init (self)

while True:
    mmt = my_memory_test()

This script kills machine perfectly too.

As Python bindings for Midgard depends on GObject bindings I started to invent own destructor. Finally solution is plain and simple. Instead of derived destructor (the one assigned to tp_dealloc), I added new one:

void
_py_midgard_gobject_destructor (PyObject *self)
{
    PyGObject *pygobj = (PyGObject *) self;
    if (pygobj->obj && G_IS_OBJECT (pygobj->obj))
            g_object_unref (pygobj->obj);

    PyObject_GC_Del (self);
}

Now the question is: does derived class destructor inherits parent class' one if explicitly defined as none? Basic debug statements clearly showed that parent's destructor is invoked but why memory usage grew till death?

No more multilang in Midgard2

Piotr Pokora — 2009-09-18T11:30:14+00:00

In brief: I dropped multilang features from Midgard2 core today. Next week, I'll drop sitegroup's ones, which means the whole architecture is going to be simplified a lot. But do not worry, we - Midgard developers - have better plan for missed sitegroups and multilangs : contexts :)

Some vala for Midgard2

Piotr Pokora — 2009-09-10T08:53:33+00:00

Recently I decided to start some Midgard2 developing work using vala. I needed to implement simple GLib's GKeyFile wrapper, so vala seemed best choice.

First of all, vala is perfect (and sexy excellent) for boring and time consuming routines like GObject class' macros and standard hooks initialization. But with real, non standard code I started to hit the wall.

GLib.KeyFile

kf.set_comment (group, key, comment);

generates:

g_key_file_set_comment (self->priv->_keyfile, group, key, comment);

Obviosuly error pointer (or NULL) is missed so code won't compile.

kf.get_groups ();

generates:

return (_tmp1 = g_key_file_get_groups (self->priv->_keyfile, &_tmp0), *result_length1 = _tmp0, _tmp1);

&_tmp0 is expected to be size_t so code doesn't compile too.

Those are minor issues as you can always edit autogenerated code and fix bugs. But this is not the point.

GLib.List

GLib.List list = new GLib.List<string> ();
list.prepend (some_string_var);

generates code which doesn't compile cause of self->priv->g_dup_func missed member.

string

I created string array property, like this:

private string[] cfgs;

and constructor:

this.cfgs = new string[0];
this.cfgs = {};

and added more elements:

this.cfgs += some_string_var;

Code compiles and runs but greatly segfaults with 'double free or corruption'

I am using valac 0.5.7-1 on 64bit Ubuntu 9.04. Might be that all issues are on mine side.

Updated:

var arr = new GLib.Array<string> (false, false, (uint)sizeof (string));
arr.append_val (abspath.ndup(abspath.length));

doesn't compile:

error: lvalue required as unary ‘&’ operand

and C code:

g_array_append_val (arr, _tmp7 = g_strndup (abspath, (gsize) string_get_length (abspath)));

Qaiku as Mojito service on moblin

Piotr Pokora — 2009-07-30T19:22:14+00:00

Just started to add Qaiku as new Mojito service for Moblin:

I used kvm alpha2-20090311 image, and (unfortunately) haven't been surprised.

Had to install many (plenty of) dev(el) packages to start some work
Spent some time trying to find out how to install gconf, cause package is named Dbus-Gconf. Uh?
No apt-get! (yes, this is personal ;)

When I added Qaiku service, which is available in 'Web services' configuration, I started to wonder how could I start testing it before adding more code. The toolbar is available in Moblin beta release, but it's not in SDK image? Does it mean I should write code using SDK image, package it, install on other device with Moblin installed and test it at last?

So far I understood I should initialize another window manager, but mutter (no idea if it's the correct one) created two squares on my image screen - white and blue one.

Also it would be very nice if Mojito developers could add more authentication types support. You can use traditional username/password authentication for Qaiku but you can use an APIKey as well. I want Qaiku service to use APIKey, so had to define 'username' in Mojito's service, but this one as you see on screenshot is kind of confusing. I bet 99,99999% of users will type their username instead of their unique uuid.

Midgard2 API with Gtk-doc

Piotr Pokora — 2009-07-08T19:48:57+00:00

Not, a big deal. Many Gnome libraries use Gtk-doc as the tool for API documentation. Midgard is just another one. For long, long time I used Doxygen, but this wasn't good enough, and there are few Gtk-doc features I like very much.

Deprecated symbols are marked nice way in search panel.

All libraries in one place.

Quick access to new symbols.

Consequence and consistence , part 2

Piotr Pokora — 2009-07-08T15:36:05+00:00

Zend is from Mars and PHP is from Moon.

This works:

zend_class_entry **ce;
const gchar *rmn = "reflectionmethod";

if (zend_hash_find(CG(class_table), (gchar *)rmn, strlen(rmn)+1, (void **) &ce)  == FAILURE)
    return NULL;

And this segfaults badly somewhere in Zend internals:

zend_class_entry **ce;
const gchar *rmn = "ReflectionMethod";

zend_lookup_class((gchar *)rmn, strlen(rmn), &ce TSRMLS_CC);

I wOndER wHere I mAde misTAkE?

Midgard2 and database transactions

Piotr Pokora — 2009-07-07T11:34:02+00:00

Mjolnir has a very nice feature, which many Midgard developers has been waiting for, for long time. Introduced midgard_transaction class provides an easy way to manage database transactions. I was wondering if transaction support should be implemented as midgard_connection methods, but I decided not to abuse connection's API and transaction class as standalone one would emit (in a future) additional signals for applications' callbacks.

midgard_transaction usage is very easy:

t = midgard.transaction()
t.begin()
t.commit()

The same API works in C, PHP and also in Objective-C Midgard bindings. Note that Midgard core itself never uses transactions. It's up to the application, whether begin, commit or rollback transaction.

I also found some issue related to SQLite transaction's identifier. If you think it could be solved better way, let us know.

Midgard2 and database views

Piotr Pokora — 2009-07-06T12:04:27+00:00

Midgard2 provides flexible support for database views. The basic idea originates from well known MgdSchema technology introduced in Midgard 1.7. And to get better overview, you probably should read midgard_view mRFC, but I'll try to explain how you could start with views quickly.

First of all, you have to create view xml file which describes and defines real database view. As view can be complex, you can also define any database join type:

<?xml version="1.0" encoding="UTF-8"?>
<Schema xmlns="http://www.midgard-project.org/repligard/1.4">
    <view name="mystyleview" table="style">
        <property name="pagename"     use="midgard_style:name" />
        <property name="parentid"        use="midgard_style:up" />
        <property name="elementname"  use="midgard_element:name" />
        <property name="value"        use="midgard_element:value" />
        <join type="left" class="midgard_host" >
            <condition left="midgard_host:style" right="midgard_style:id" />
        </join>
        <join type="inner" class="midgard_element" >
            <condition left="midgard_element:style" right="midgard_style:id" />
        </join>
        <join type="inner" table="element_i" >
            <condition left="midgard_element:sid" right="midgard_element:id" />
        </join>
    </view>
</Schema>

Once xml file for view is created, it should be copied to views directory under Midgard2 share dir. This one depends on midgard core prefix, and in my case it's /usr/share/midgard-2.0/views/. Then, you can start any Midgard2 application. View definition will be read on start up, and new GObject class will be created on the fly. If your application is written in PHP or Python, then proper language related class will be initialized as well.

Now, you can create new mystyleview instance (in Python):

mystyle = midgard.view.mystyleview()

or in C:

MidgardView mystyle = g_object_new(g_type_from_name("mystyleview"), NULL);

OK, but at this moment, you can do nothing with your view. Database view is considered read only, so you need to fetch some database records, which are represented as objects.

First of all, prepare database storage for newly created view. MidgardStorage (midgard_storage) should do the hard job for you:

midgard.storage.create_class_storage("mystyleview")

Of course, you can delete this storage, once you decide you do not need it any longer:

midgard.storage.delete_class_storage("mystyleview")

Both tasks can be done also in C or in PHP.

Now, when database view is created you can start query objects. The best helper, is MidgardQueryBuilder:

qb = midgard.query_builder("mystyleview")
qb.add_constraint("pagename", "<>", "")
list = qb.execute()

As midgard_view is nothing but database storage abstraction, you can also benefit from midgard_collector or midgard_reflection_property.

Also, any instance of registered midgard_view class has properties which has been defined in xml view file. So (for example) access to pagename property is done via list[0].pagename.

Midgard2 - new epoch

Piotr Pokora — 2009-04-29T10:10:02+00:00

Today, we made first stable release for the new Midgard2 generation. For years, Midgard focused on PHP and well known LAMP platform. With Midgard2 we created completely new architecture based on stable and widely used libraries:

GLib
GObject
LibGDA
LibXML

And from pure web environment our target became: desktop, mobile devices and web as well. With Midgard2 you can select the language you would like to write your favorite application:

C
Python
PHP
Objective-C

Binary packages will be available soon at OBS repository. And later on, you can expect them also in maemo extras.

When MySQL goes away

Piotr Pokora — 2009-03-18T09:47:17+00:00

I am debugging Midgard problem related to MySQL 2006 error. It's well known as "MySQL server has gone away". What I found so far, is an easy way to set very short timeout for debugging purposes. I execute such query in MySQL shell (and it sets timeout for 15 seconds):

set global wait_timeout = 15;

Similarly, you set increase yout timeout if sercer times out too quickly:

set global wait_timeout = 86400;

This one sets timeout for 24 hours.

Once you do so, you can notice different values when executing show variables query. Do not worry, the query you used, returns variables from the session you are just running.

Consequence and consistence

Piotr Pokora — 2009-02-12T18:01:49+00:00

I tell it here and there. But people tend to forget, so it's better to write it down. I know, some people (developers) already don't like me, but it's not really my fault they write bad code. And since now, I will (consequently) write about something they produce. Ah! It's PHP or Zend Engine 2, if I forgot to mention.

Let's look at one file, php_reflection.c from reflection extension.

/* Invoke the __toString() method */
ZVAL_STRINGL(&fname, "__tostring", sizeof("__tostring") - 1, 1);

So what is the method name? __tostring or __toString? What minimal alphabet they use?

And note second line:

ZVAL_STRINGL(&fname, "__tostring", sizeof("__tostring") - 1, 1);

And one more from the same file:

zend_hash_find(EG(function_table), lcname, name_len + 1, (void **)&fptr);

and one more:

zend_hash_update(Z_OBJPROP_P(object), "name", sizeof("name"), (void **) &name, sizeof(zval *), NULL);

What's the deal with those three lines of code? API is created such way, so it forces you to pass length of the passed string. So if you look at examples above, how do you count length of the string?

midgard_datetime

Piotr Pokora — 2009-01-22T16:39:20+00:00

I started to work on midgard_datetime class. Pure PHP class which should represent any property of Midgard MgdSchema object which holds datetime string. A pure, means that it focuses on PHP language only, and in fact, it's implemented in midgard extension.

Why midgard_datetime? And not just simple DateTime class, as suggested in related ticket? Answer is simple. PHP has limits. Imagine, every datetime property is propagated as DateTime object. And when you want to read it, you simply invoke format() method. But what to do when you need to change date or time? You invoke setDate() method and what next? Did underlying GObject's property value has been changed as well? No. There's no features like signal emission or simple event notification on PHP level. So, even if you write own callback, no single line of code will trigger it (As a decent framework Midgard supports events).

I wrote midgard_datetime class (DateTime derived) with own underlying C structure and assigned own hooks for it. All in all, at some point I need to know what is the object and what is the property a midgard_datetime holds value for. And this failed because DateTime objects hold own underlying C structures, so when all Zend OO related hooks return void type, you never know what you really get: DateTime structure, midgard_datetime one or maybe mather-in-law? Implementation simplified. But it doesn't mean it became more elegant. I had to register private properties. Yes, this is common for PHP code, but on C level you have armory of possibilities, and had to forget about it.

Examples

In Ragnaroek, you used to print datetime property:

echo $object->metadata->created;

and in vinland, what you wanted to print is an object, so you need to invoke its method:

echo $object->metadata->created->format("c");

midgard_datetime is automagically initialized with DateTimeZone object, so if you need timezone you invoke getTimezone method. Of course, the only one correct timezone for Midgard datetimes is UTC. And you can not change it.

For more flexibility, every DateTime (or derived one) class' object is converted to GValue which holds ISO 8601 formatted string, so ot only objects are "affected" by such conversion. You can pass such objects as Query Builder constraints' values:

$md = new midgard_datetime();
$md->setDate(2009, 01, 02);

$qb = new midgard_query_builder("midgard_article");
$qb->add_constraint("metadata.published", ">", $md);

Performance

I made two simple tests. Both have a loop (100 steps) and assign string datetime. First one uses:

$a = new midgard_article();
$b = $a->title;

and takes from 0,0060 to 0,0080 seconds. This test is almost the same as assigning datetime property the "old" way.

Second one uses midgard_datetime:

$a = new midgard_article();
$d = $a->metadata->created->format("c");

and takes from 0,0180 to 0,0230 seconds.

Looks like midgard_datetime implementation is up to 4 times slower. But as it extends DateTime, there's many features implemented "for free". And if you take into account, that first test requires additional function calls, like strtotime() and date() (to format output), midgard_datetime implementation will be up to 2 times slower. And I think it's not a big problem. midgard_datetime object is created on demand, only when you need to read property. And in practice, you do not need thousands of datetimes available at the same time.

Issues

MySQL doesn't work with constraint used in QB's example. It simply treats string verbatim, so SQL part looks like this:

WHERE metadata_published = '2019-01-01T00:00:00+00:00'

It works, if you remove timezone offset. So I believe fixing the issue is a matter of date formatting.

Also there's no possibility to convert datetime values (returned by collector for example) to midgard_datetime objects. Such values are initialized as GValue values of string type. And this should be fixed by wider midgard timestamp values usage in core.

You can install latest packages, if you would like to try those.

ZEND_ARG_INFO reflections in midgard php bindings

Piotr Pokora — 2009-01-16T12:42:20+00:00

Yesterday I finished ZEND_ARG_INFO implementation support in PHP bindings for Midgard. I am happy as I closed another ticket, so this nice feature will be also available for Vinland. Yes, I must say this is nice feature. However, at first I was very unhappy how it's implemented in ZE2.

For simple class implmentation you may use ZEND_ARG macros, quite simply.

midgard_connection::open($name):

ZEND_BEGIN_ARG_INFO_EX(arginfo_midgard_connection_open, 0, 0, 1)
    ZEND_ARG_INFO(0, name)
ZEND_END_ARG_INFO()

But for dynamic classes, like Mgdschema ones, things are more complicated.

get_by_guid($guid) method:

{ NULL, 0, NULL, 0, 0, 0, 0, 0, 1 },
{ "guid", sizeof("guid")-1, NULL, 0, 0, 0, 0, 0, 0 },

Especially, if you take into account fact, that method's arguments are defined with the same structure as method itself. And of course (as usual), you can not read about this in ZE2 documentation. Just investigate and hack sources.

Anyway, what does it mean for Midgard? Better reflection and nice (almost perfect) way to introspect all Midgard core API via PHP classes.

Let's look at simple example (I used the same code, provided in PHP docs):

$method = new ReflectionMethod('midgard_user', 'password');
$p = $method->getParameters();

foreach ($method->getParameters() as $i => $param) 
{
    printf(
    "-- Parameter #%d: %s {\n".
    "   Class: %s\n".
    "   Allows NULL: %s\n".
    "   Passed to by reference: %s\n".
    "   Is optional?: %s\n".
    "}\n",
    $i, // $param->getPosition() can be used from PHP 5.2.3
    $param->getName(),
    var_export($param->getClass(), 1),
    var_export($param->allowsNull(), 1),
    var_export($param->isPassedByReference(), 1),
    $param->isOptional() ? 'yes' : 'no'
    );
}

And the output:

-- Parameter #0: username {
    Class: NULL
   Allows NULL: false
   Passed to by reference: false
   Is optional?: no
}
-- Parameter #1: password {
   Class: NULL
   Allows NULL: false
   Passed to by reference: false
   Is optional?: no
}
-- Parameter #2: hashtype {
   Class: NULL
   Allows NULL: false
   Passed to by reference: false
   Is optional?: yes
}

A real cool thing about such reflection is possibility to create one unified documentation system for code written purely in PHP and for that one written in C extension. If Zend introspection will allow this, we should have much better documentation with less amount of work. And this will be first cool thing provided by Zend :)

Always blame...

Piotr Pokora — 2008-12-09T11:53:22+00:00

I must say I like to read threads like this. Yes, for me it's kind of endless pleasure to read every single sentence and word in such thread.

Especially this one:

there was a known memory corrupt issue in 5.2.5

Faster MidCOM static files.

Piotr Pokora — 2008-11-27T20:38:18+00:00

Better late then never, but just today I realized that MidgardEngine directive from Midgard apache module can not work inside Directory one. Not because module has a bug. But because with midgard apache module, apache won't be able to check if it's serving file from such directory. It is the main purpose of midgard module. Serve data from database, not from filesystem.

Of course, MidgardEngine will work in virtaul host scope. But in directory scope -will never work.

How to set MidgardEngine to Off for specific directory then? Use Location. I configured it simply for my virtual host:

<Location /midcom-static/>
    MidgardEngine Off
</Location>

Sauna, suana, sauna or Midgard Gathering

Piotr Pokora — 2008-11-12T20:24:26+00:00

I was to blog earlier but this issue made me busy. On monday, me and solt came back to Poland from Midgard Gathering. Second gathering this year. I must say I think I am getting older as it's getting hard to focus when there are so many people and sauna time ends very late night :)

Bergie's blog describes gathering very nice and with more details.

Anyway, few very interesting points:

Midgard and Vala
PHP and Python bindings communication
Almost successful Midgard build on windows
Midgard Foundation!

And as Vinland requirements clarified a lot, I am looking forward to see Midgard2 as a gateway for Gnome to web servers and services world.

ap_get_module_config and RHEL5

Piotr Pokora — 2008-11-12T19:18:22+00:00

This is an issue. I couldn't find why Midgard apache module refused to work on RHEL5.It just started to segfault. What's even more interesting, it started to segfault when trying to access read only data provided by apache. If you are familiar with apache modules' internals you probably know that all routines are registered as hooks. Even that ones which holds server and directory config. In normal case you register hook and get configuration later, when it's needed.

Just like this:

midgard_directory_config *dcfg = 
    (midgard_directory_config *)ap_get_module_config(r->per_dir_config, &midgard_module);

I found these two posts quite interesting. First because it affected also midgard-apache module few years ago, and the second because it clearly shows there something odd happens. Why odd? Because you never sets directory or server config explicitly. You just register function and it's up to the server implementation when it's invoked.

In my case ap_get_module_config(r->per_dir_config, &midgard_module) returned NULL all the time. So I added debug messages which showed that server and directory configuration hooks are invoked. Spent plenty of time trying to figure it out.

Your hooks are working very good (the same code works with older version of Midgard)
Hooks are invoked. For sure.
ap_get_module_config returns NULL.

Three facts which made me think it's something wrong with Apache. Not the module itself. And after many trials and errors, I found guilty AM_CFLAG: -D_FILE_OFFSET_BITS=64. Also interesting fact, this flag is able to slow down PHP itself.