Piotr Pokora

When metadata creator is empty

Piotr Pokora — Fri, 23 May 2008 13:49:48

Quite major bug has been found today. 1.8.7 Midgard release doesn't populate metadata_creator guid. Midgard news page has a note about this.

I wrote additional script which should help with database upgrade if needed:

#!/bin/sh

DB="midgard"
MYSQLCMD="/usr/bin/mysql"

THEADER="Tables_in_${DB}"
MIDGARD_TABLES=`$MYSQLCMD -D $DB -e 'show tables' | grep -Ev '(_i\s*\|?\s*$)|repligard|sitegroup|Tables_in_.+'`

for TABLE in $MIDGARD_TABLES; do

    QUERY="UPDATE ${TABLE} SET metadata_creator=REPLACE(metadata_authors, '|' ,'')  WHERE metadata_creator='' AND metadata_authors<>'' AND metadata_authors<>'||' AND LENGTH(metadata_authors) < 39"
    echo "Fixing table: ${TABLE}"
    $MYSQLCMD -D ${DB} -e "${QUERY}"

    echo "Looking for multiple authors"
    QUERY="SELECT guid from ${TABLE} WHERE LENGTH(metadata_authors) > 38 AND metadata_creator='' AND metadata_authors<>'||' AND metadata_authors<>''"
    $MYSQLCMD -D ${DB} -e "${QUERY}"

done

Voices in my head

Piotr Pokora — Mon, 19 May 2008 22:15:37

It was last sunday. And there are still "Voices in my head". Riverside guys made real show in best studio in Poland, and it's quite obvious as they recorded live DVD at the same time. Probably this DVD is going to be in stores this winter. Damn! So long time.

First time I heared them in LizardKing pub in Łódź. It was February, this Year. Since then, I am addicted. There is no single word that can be said about this event. No one seems to want to comment it. Only few of them appears from time to time on different forums: perfect.

"Join in the shadow dance"

PEAR registry, where are you?

Piotr Pokora — Mon, 19 May 2008 12:19:13

I wrote this simple script to clear my local pear setup and to have clean pear instalation for Midgard.

apt-get remove php-pear
rm -rf /usr/share/php/PEAR
rm /root/.pearrc
rm -rf /etc/pear
rm -rf /tmp/pear
apt-get -y install php-pear

So, when I cleaned my pear ,imagine, how much confused I was when I saw this in my terminal:

MIDGARD SETUP: Role_Web - installed: 1.1.1 available: 1.1.1

wtf?

pear list-files pearified/role_web
Installed Files For pearified/role_web
======================================
Type Install Path
php  /usr/share/php/PEAR/Installer/Role/Web.php
php  /usr/share/php/PEAR/Installer/Role/Web.xml
php  /usr/share/php/Pearified/Role/Web/setup.php

ls -l /usr/share/php/PEAR/Installer/Role/Web.php
ls: /usr/share/php/PEAR/Installer/Role/Web.php: No such file or directory

PEAR registry...where are you?

Updated: I found registry and channel files under /usr/share/php. So on debian these two lines must be added additionally:

rm -rf /usr/share/php/.registry 
rm -rf /usr/share/php/.channels

MySQL doesn't support TEXT type

Piotr Pokora — Thu, 08 May 2008 14:46:56

Yes, it does not. But you could say:

"oh no, Piotras, it does, please read this part of manual"

Oh, yes indeed. They mention "The four TEXT types", but it doesn't seem to help me even a bit. Why? Bergie and Solt reported quite weird issue. Text value ( text type in database ) has been returned as array on PHP bindings for Midgard.

I started to investigate it from the very beginning, to catch why text type becomes an array:

Parameter class has value member of longtext type.
Database column for this property has longtext column type.
GDA library with MySQL provider uses MySQL column types to determine what value should be initialized on C level.
GDA gets BLOB type.
GDA initializes its own GdaBinary type( GdaBinary is GBoxed derived type ).
PHP bindings for midgard and especially GValue and zval converter gets boxed type.
We support only array type of gboxed one so zend array is initialized and returned.

Not easy to fetch at first glance, isn't it?

I was wondering and checked GDA sources ,and indeed MySQL provider implementation is not aware of something like TEXT or LONGTEXT. Later on I checked MySQL's headers and supported column types.

So, there is support for strings (char and varchar):

MYSQL_TYPE_VAR_STRING
MYSQL_TYPE_STRING

And for blobs:

MYSQL_TYPE_TINY_BLOB
MYSQL_TYPE_MEDIUM_BLOB
MYSQL_TYPE_LONG_BLOB
MYSQL_TYPE_BLOB

And what about TEXT and LONGTEXT? And what about those 4 mentioned text types? Nothing. There's completely nothing. How can I know if retrieved data is binary or text type then? Use randomized type? Maybe for you blob type is the same as longtext, but for me plain text book stored in field is quite different thing than compressed image file.

Midgard with Lighttpd

Piotr Pokora — Tue, 22 Apr 2008 08:05:55

Please. Read the whole this simple howto. It's adressed for developers who would like to have Midgard and Midgard2 available as web services on the same machine. Of course, you may still install or compile Apache 1.3 as the second web server.

First of all you need to install and configure Lighttpd web serwer. On debian I installed it using apt-get:

apt-get install lighttpd

Do not forget to stop Apache before installing lighttpd. If you do not, dpkg will return with failure status as lighttpd tries to bind to port 80 by default when package is installed.

Also, you need to have php5 compiled with fast-cgi support. Again, on debian install it quickly:

apt-get instal php5-cgi

Edit cgi's php.ini file (/etc/php5/cgi/php.ini), and add midgard support:

extension=midgard2.so
midgard.configuration="midgard"
midgard.http="on"

Then configure your new web server. I found these two sites very helpful to configue lighttpd server:

Later on, you need to install Midgard2 from trunk, which is going to be released as alpha2 soon.

In my case, I configured lighttpd with these options:

server.port = 81

Just to have both, Apache and lighttpd available. You may of course use different port if you like.

Enable required modules, mod_rewrite and mod_fastcgi:

server.modules              = (
        "mod_access",
        "mod_alias",
        "mod_accesslog",
        "mod_compress",
        "mod_rewrite",
        "mod_fastcgi",
)

Configure fastcgi:

fastcgi.server = (
    ".php" => (
            "localhost" =>
                    (
                            "socket" => "/tmp/php-fastcgi.socket",
                            "bin-path" => "/usr/bin/php5-cgi"
                    )
            )
    )

Configure virtual host:

$HTTP["host"] == "myhost" {
    server.port = 81
    server.document-root = "/var/lib/midgard/vhosts/myhost/8080"
    url.rewrite-once = ( "^(.*)\.*" => "midgard-root.php" )
}

I use virtual host on 8080 port with Apache, so I wanted to re use its DocumentRoot. You will change it of course, depending on your needs.

midgard-root.php file should be copied or symlinked from midcom's git repository. I do not have to remind that midcom itself should be installed and ready to use.

Do something you should never do in real life. It seems that lighttpd drops root privileges too quickly and php module is loaded with configured user and group privileges. At least I couldn't manage to open root's configuration files from system configuration directory( which is easily done with Apache ). To establish database connection(s) you must have all secret configuratoin readable for the whole world.

chmod 644 /etc/midgard-2.0/conf.d/*

After this, restart lighttpd:

/etc/init.d/lighttpd restart

And open http://myhost:81 with your favourite browser. If all went fine, you should have midcom's web page without style. Reason is that we do not make midcom-static exception in our hosts' rewrite condition.

MidCOM3 without apache module

Piotr Pokora — Fri, 11 Apr 2008 14:15:07

This is really HOT!

It's proof of concept to work with Midgard without its Apache module. And the concept is cool :) Especially with this lovely MidCOM3 screenshot.

Basics of midgard_dbus

Piotr Pokora — Tue, 08 Apr 2008 10:55:41

midgard_dbus API is very simple, but my approach was to make it just simple as possible. Basically we have two major methods of midgard_dbus:

constructor, which registers object at path
send, which asynchronously send message to object at given path

New initialized object doesn't do anything special. It just waits for any call to him and then emits "notified" signal, which you can connect callback to.

I tested simple code with midgard-php and midgard-python.

midgard_python "service":

import dbus.mainloop.glib
import _midgard as midgard

def mbus_callback(object, arg):
        print "Hi! I am midgard_dbus from midgard-python. I got message:"
        print object.get_message()

mbus = midgard.dbus("/midgard_article")
mbus.connect("notified", mbus_callback, "foo")

mainloop = gobject.MainLoop()
mainloop.run()

midgard-php "client":

$message = "Greetings from midgard-php!(" . mgd_version() . ") PHP ver." . phpversion();
midgard_dbus::send("/midgard_article", $message);

I started php script which immidietialy ended, and on midgard-python service's terminal I got this message:

Hi! I am midgard_dbus from midgard-python. I got message:
Greetings from midgard-php! (2.0alpha0) PHP ver.5.2.5-3

Probably the code will be soon ported to 1-9 branch, so I will be able to test midgard_dbus in Apache environment. But for now, basic idea looks fine and works fine. We just notify objects at path with particular message, and local proccess does the rest with connected callbacks.

D-Bus and Midgard

Piotr Pokora — Fri, 28 Mar 2008 12:40:12

Midgard is decent CMS framework. Not just yet another PHP based CMS. And as every decent piece of code must have many features. I just added initial test cases with D-Bus support. They use midgard-python and dbus modules.

Usage is simple. You start service as one proccess and client as another one. If all goes fine, client notifies daemon that some object needs to be imported to database, then listener service imports this object to database. Listener is not aware who sent such data and even not aware what kind of language was used to send notify.

So in theory ( it's not yet API implementation ) application written in PHP and using Midgard 1.9 may may send notification and listener may be daemon writen in python using Midgard 2.0. Even on the same machine.

Check initial test code!

midgard-python once again

Piotr Pokora — Tue, 18 Mar 2008 21:02:12

Bergie just blogged about initial MidCOM3 architecture. He included nice and very easy example of midgard-php API. And while he mentioned some python based solution I decided to add short and working example of midgard-python in action.

All in all, it's good to have, especially now, when Midgard2 is already an alpha and available also as packages for maemo.org platform.

Gnomegard

Piotr Pokora — Sun, 09 Mar 2008 12:46:58

I think t's going to be sexy. Very sexy.

4 years ago we planned some features.
Few weeks ago world heard about Midgard2 first alpha release.
Gnome 2.22 should be released soon.
Gnome 2.22 should contain new API for Virtual File System.

I must test it! mgd:// uri will look lovely :)

Midgard2 and Gtkdoc

Piotr Pokora — Thu, 06 Mar 2008 20:32:00

One day I realized that for last few months, I managed to fix (few times) some documentation part for Midgard core and PHP bindings. Those fixes almost always included properties of some class. And recently I started to wonder how to write documentation for events in midgard, once. Well, not about the idea, but at least I wanted to avoid writing ( few times in few different places ) what events some class provieds.

Since Midgard 1.8 we used doxygen, which is very good, with cool features for writing docs, but for me ( and writing GObject based code and docs ) is not perfect enough. Mainly you can not document class inheritance, object properties or class signals. And what is more important, you can not do this automagically.

I decided to test gtkdoc ( which is also far from being perfect and seems to be missing doxygen :), and got quite nice documentation part for a start.

List of MgdSchema object events
midgard_config properties

Those are created automagically, so I do not have to document them. PHP or Python developer might always want to look for additional information in core documentation, without need to duplicate many informations in many places and without risk that those docs are getting out of date quickly.

Bad thing about gtkdoc and doxygen is the fact that docs should be located in c files ( instead of headers ) and must be reformatted a bit. But once they are written, one is able to use even devhelp to read them.

Testing Midgard 1.9

Piotr Pokora — Thu, 21 Feb 2008 14:44:58

It's not released yet, but I added few scripts which build debian packages hourly, so every developer can install it easier. To make things even more easier I tested this simple script which does everything:

MIDGARD_TEST_REPOSITORY="deb http://www.midgard-project.org/nightly/debian ./"
APT_SOURCES_LIST="/etc/apt/sources.list"

if ! grep $MIDGARD_TEST_REPOSITORY $APT_SOURCES_LIST; then
        cp ${APT_SOURCES_LIST} ${APT_SOURCES_LIST}.midgard.bak
        echo "Adding ${MIDGARD_TEST_REPOSITORY} to ${APT_SOURCES_LIST}"
        echo ${MIDGARD_TEST_REPOSITORY} >> ${APT_SOURCES_LIST}
fi

echo "Updating packages info..."
apt-get update

echo "Installing midgard..."
apt-get --reinstall install libmidgard10 php5-midgard libapache2-mod-midgard midgard-common

Of course you should be logged in as root or at least run it with sudo. As the last line you can add:

/usr/sbin/datagard

midgard-schema and database update

Piotr Pokora — Mon, 11 Feb 2008 13:21:52

I ported some minor database routines from Midgard branch-1-9 to stable branch today. It doesn't mean I ported unstable features. On the contrary , I fixed quite important issues: #193,#214 and #78.

What does it mean exactly?

Datagard creates tables and adds some additional columns and indexes but only those mandatory like sitegroup or guid. midgard-schema is improved and make all database update more clever way:

midgard_reflection_property is used to check if property should be indexed ( like parent property or link one )
Indexes are created for commonly used metadata columns

When update is performed, midgard-schema checks every single table and column if anything should be done with it:

Update is made as much faster as many unchanged columns you have in table ( comparing to shell's datagard )
During developing, you may alter table and change columns' type "on the fly".

midgard_blob once again

Piotr Pokora — Thu, 07 Feb 2008 18:56:38

This class has been introduced in Midgard 1.8 branch, but mostly to satisfy replicator. Today I finished experimental API for Midgard 2.0 and Midgard 1.9 ( with PHP language bindings as well ).

Documentation for midgard_blob needs many improvements but you can take a look at it already today.

Completely new features which wait for you in Midgard 1.9 include:

You may create attachment for another attachment
You may share attachments among many objects
You may use any PHP API part which requires file's absolute path or file handler.
You may create new files quickly

When midgard_setup fails

Piotr Pokora — Tue, 22 Jan 2008 21:17:27

I have been multitasking a bit and talk with Jerry about erlang issues and ran new datagard in the background. Well... it's automagic setup so it's quite clever and can make few things for you. Without any needless questions.

So, what I got when I switched to terminal again?

MIDGARD SETUP: Sitewizard created host successfully. Now cleaning up... 
MIDGARD SETUP: Preferred pear package state changed ( beta ) 
MIDGARD SETUP: Reloading apache http server 
MIDGARD SETUP: Please wait... 
Warning: file_get_contents(http://ppmobile): failed to open stream: HTTP request failed! HTTP/1.0 500 Server Error
MIDGARD SETUP: ! WARNING ! : Can not request newly created host ( http://ppmobile )  
MIDGARD SETUP ERROR: Please, analyze apache server log files 
Quitting...

Nice, now I need to analyze apache's log file :) Ah, keep in mind that all these messages are much more detailed in written log file. But now I do not need it. I will use later when I miss terminal screen.

Memory corruption in midgard-php

Piotr Pokora — Fri, 18 Jan 2008 10:00:11

Just two days ago I blamed Suhosin patch, and this morning I found probably the source of the problem.

Log entry is clear:

PHP Fatal error:  Cannot instantiate abstract class \x85\xd7\xe9\x84\x1d

I found that mgd_get_object_by_guid is not aware of Zend2 (PHP5), and code "focuses" on Zend1 (PHP4). I can not say that code uses ZE1 or ZE2 API. The issue is hard to find and it's never noticed when code is being compiled.

If you need to write code for ZE1 and ZE2, keep it in mind:

ZE1: zend_class_entry *ce;
object_init_ex(return_value, ce TSRMLS_CC);

ZE2: zend_class_entry **ce;
object_init_ex(return_value, *ce TSRMLS_CC);

But when you look for class pointer you always pass (void **) &ce) as zend_hash_find parameter. So when you use ZE1 class pointer and code is being compiled against ZE2, you get no single notice or warning. And it's very easy to omit such case, especially that code itself runs almost correctly. Almost means that sometimes it fails.

I will make 1.8.6 release next week because the issue is very serious. Meantime if you used to notice random Segfaults in error.log, please change your code and use ( recommended anyway ) midgard_object_class::get_object_by_guid instead of mgd_get_object_by_guid.

Suhosin Patch

Piotr Pokora — Wed, 16 Jan 2008 10:10:03

I have been asked to look at random segfaults which started to appear after Midgard 1.8.5 upgrade. I checked many places and was wondering what really triggers them.

Fortunatelly I was able to run Apache through gdb and was much more wondered. Backtraces pointed to ( randomly ) : xdebug, eaccelerator, php5 library itself. Even Fatal Error like "Cannot redeclare class" caused segfaults.

So, I was completely out of ideas, when I noticed this one in Apache's log file:

PHP/5.2.4-0.dotdeb.0 with Suhosin-Patch

Suhosin, the one I started to blame. And looks like I was right. Server is perfectly fine when php5 has been replaced with the one without Suhosin Patch.

I did google a bit and found how many people reported segfaults with PHP5 with Suhosin Patch. However Suhosin itself is not blamed for segfaults I think I should ask few questions:

If I have at least 5 extensions loaded , which one triggers segfault if Suhosin is enabled ?
Which one is badly written if PHP doesn't segfault without Suhosin?
If one extension is badly written where can I find documentation how it should be written well to work fine with Suhosin?
Who made detailed tests and is sure that all extensions corrupts memory? All but Suhosin ?

When you think you are drunk

Piotr Pokora — Sat, 12 Jan 2008 11:12:16

I haven't been writing about new features of Midgard for the last 3 weeks, but it's high time to stop write about new ones and instead write about new upcoming releases ( which I am tuning recently ). So I tested and built release packages wherever I could, including maemo scratchbox. Unfortunatelly I found python-midgard package buggy... so I took a closer look at python, and here's what I found:

ls -l /usr/bin/python*
lrwxrwxrwx  1 root root    9 Feb 11  2007 /usr/bin/python -> python2.4
-rwxr-xr-x  1 root root 3384 Jun 14  2006 /usr/bin/python2.4
-rwxr-xr-x  1 root root  119 Jun 14  2006 /usr/bin/python2.4-pyrexc

Ok, nothing unusual, right? python command is just a symlink to python2.4. Which is cool of course, isn't it?

But strace wrote weird logs so I decided to check python version:

/usr/bin/python2.4 -V
Python 2.4.2
/usr/bin/python -V
Python 2.3.4

Please, please tell me why the hell command invoked via symlink returns different version.

midgard_user_auth

Piotr Pokora — Mon, 17 Dec 2007 12:11:41

I started to make new Midgard2 auth features backward compatible. So, to make things smooth and transparent I added new internal function which makes legacy authorization and it's triggered by simple check, check if new midgard_user table exists in database.

The good start:

database is created with midgard branch-1-9
python is used as scripting language
midgard2 library is used with test

And simple example code:

user = midgard.user.auth("admin", "password", "python-sg", False);
if(midgard.user.is_admin(user)):
    print ("User is an admin")

Output with method names info:

** INFO: user.is_admin(...)
User is an admin

I didn't make any database changes, so let's hope both midgard versions could be used together soon.

Beauty and the Beast

Piotr Pokora — Tue, 04 Dec 2007 21:55:59

I am not sure which one is the beast and which one the beauty. But I think both midgard-python and midgard-core may be both :)

Python seems to be the second Midgard's scripting language now. Two weeks ago I wrote about initial usable bindings work, and today there's much more:

Midgard Query Builder
Midgard Collector
Midgard Object Class ( pseudo class for MgdSchema classes introspection and reflection )
Midgard Reflection Property
MgdSchema objects' parameters

Midgard Replicator is not yet fully implemented for midgard-python because it simply doesn't exist in Midgard2 ( aka trunk ). Reason why it hasn't been moved there is fact that replicator uses language specific SQL queries while Midgard2 itself uses something which you could name "built in" languages support. Whichis nothing more like languages persistant cache in details.

You can take a look at midgard-python test examples and compile module if needed. And I think I should start count days to first Midgard2 alpha release, and first python applications for gnome or maemo :)