Piotras' blog > Archive > 2006-09-01 - 2006-09-30

Piotras' blog: Archive

2006-09-01 - 2006-09-30

Midgard2 replication preview

Posted on 2006-09-02 12:43:39 UTC.


Few days ago I wrote about exporting objects from Midgard database. Now it's time to import them to other database or even the same. I can not elaborate about API method, cause it's simply and easy.

bool mgd_import_object(string xml_content);

or static method ( a bit unlogic if you want to tie to particular class ):

midgard_classname::import(string xml_content);

or midgard-core's API:

 gboolean midgard_import_object(MidgardConnection *mgd, gchar *xml);

Usage is very simple , you just pass xml content ( as serialized object's data ) as function or method parameter. Xml content is parsed in memory and particular object instance is created. Then , import method starts to figure out what's the best method which should be invoked:

Object's record identified by guid is not found:

  • if it's already purged , do nothing
  • if it's not found in repligard's table , create new record

Midgard API is used to create new record , so for example you do not have to worry about duplicated name&up pairs.

Object's record is found:

  • if imported object's revised datetime is less recent or the same as the same value of database object return with error number MGD_ERR_OBJECT_IMPORTED
  • in other case , if imported object is marked as deleted , delete database object
  • if both cases do not match update it


So , poor example could look like this: 

$filename = "/path/to/midgard_article_b531b35cf54111daa870f9b1239adb8fdb8f.xml";

$filehandle = fopen($filename, "r");

$content = fread($filehandle, filesize($filename));

midgard_article::import($content);

/* or mgd_import_object($content); */


It's very initial real Midgard2 replication feature, but it perfectly feets a future "big picture" ideas:

  • Send me this document, report. I will reuse it's content as I need.
  • Send me this news with attachment's , I will import it to my MidCOM's app and reuse with self defined parameters.
  • Make fast backup of my objects
  • Send me your last note to my jabber account ( if I am not available , it will be waiting for me)
  • Export only approved documents.
  • Import every single object I got with POST request data.
  • etc etc etc

Past years proved that replication app is always far from being perfect and what you really need is not always doable or is very complex. I hope we're going to implement powerfull and flexible replication feature ever was made for Midgard.

Generating docs with Doxygen

Posted on 2006-09-12 12:53:06 UTC.


Midgard-core documentation is generated with doxygen package. Last sunday I experimented a bit with creating MidCOM docs with it. I was wondering if doxygen is able to generate PHP documentation while it's optimized for C, C++ and Java.

First. Doxygen package includes nice wizard gui invoked with doxywizard command. So when it's running you can simply define what you really need.

  • optimize for particular language
  • create xml output , manual pages or even chm file
  • create graphical diagrams for classes and relations
  • create html ot latex
  • create html in frames or not
  • plenty more features

Second. It was big surprise to see that basic documentation for MidCOM was created in 6 minutes.

Third. It was huge surprise to see that full documentaion with graphical diagrams and source code references was created in 15 minutes.

An interesting thing is the fact that config file created by doxywizard can be easily reused for documentation generated from cron. So in such case all you have to worry about is calling doxygen with specific doxywizard configuration files.

An issue which was clearly noticable was lack of midcom application class. However midcom package has been included in documentation. Probably it's some doxygen issue or simply the fact that MidCOM docs has been created for PHPDoc , so the syntax might be a bit different for both generators. Anyway , first impression was very nice. Doxygen created very nice html pages with classes , variables, modules , diagrams, etc etc. For me personally, for the first time MidCOM docs became readable. Html pages created with PHPDoc seems to be very nice as some IDE plugin with code completion support. Not for reading anyway.

Midgard database indexes

Posted on 2006-09-22 14:00:39 UTC.


A week ago Rambo wrote about MySQL optimization tricks. Today I decided to make detailed tests with database performance.

Idea

I use midgard_query_builder methods to produce SQL query, but the main point is to test database itself so all queries are executed in MySQL prompt as setting properties from MySQL resources is a bottleneck here. 

$qb = new midgardquerybuilder("midgard_topic");

$qb->add_constraint('up', '<>', 0);

$qb->add_constraint('parameter.domain', '<>', '');

$qb->add_constraint('parameter.name', '<>', '');

$qb->add_constraint('metadata.created', '>', '2002-02-01 00:00:00');

$qb->add_constraint('metadata.published', '>', '2002-02-01 00:00:00');

Query:

SELECT topic.guid, topic.sitegroup, topic.metadata_creator, NULLIF(topic.metadata_created,'0000-00-00 00:00:00') AS metadata_created, topic.metadata_revisor, NULLIF(topic.metadata_revised,'0000-00-00 00:00:00') AS metadata_revised, topic.metadata_revision, topic.metadata_locker, NULLIF(topic.metadata_locked,'0000-00-00 00:00:00') AS metadata_locked, topic.metadata_approver, NULLIF(topic.metadata_approved,'0000-00-00 00:00:00') AS metadata_approved, topic.metadata_authors, topic.metadata_owner, NULLIF(topic.metadata_schedule_start,'0000-00-00 00:00:00') AS metadata_schedule_start, NULLIF(topic.metadata_schedule_end,'0000-00-00 00:00:00') AS metadata_schedule_end, topic.metadata_hidden, topic.metadata_nav_noentry, topic.metadata_size, topic.metadata_published, NULLIF(topic.metadata_exported,'0000-00-00 00:00:00') AS metadata_exported, NULLIF(topic.metadata_imported,'0000-00-00 00:00:00') AS metadata_imported,topic.metadata_deleted, topic.metadata_score, topic.up AS up,topic.id AS id, NULLIF(topic.created,'0000-00-00 00:00:00') AS created, topic.revisor AS revisor, topic.score AS score, topic.code AS code, topic.creator AS creator, NULLIF(topic.revised,'0000-00-00 00:00:00') AS revised, topic.name AS name, topic.revision AS revision, topic.owner AS owner, topic_i.title AS title, topic_i.lang AS lang, topic_i.description AS description, topic_i.sid AS sid, topic_i.extra AS extra FROM topic_i,topic,record_extension WHERE (topic.up <> 0 AND record_extension.domain <> '' AND record_extension.name <> '' AND topic.metadata_created > '2002-02-01 00:00:00' AND topic.metadata_published > '2002-02-01 00:00:00' AND topic.name <> '') AND topic.id=topic_i.sid AND topic_i.lang IN (0, 0) AND topic.guid = record_extension.parent_guid AND topic.guid = record_extension.parent_guid AND topic.sitegroup IN (0, 0) AND topic.metadata_deleted = FALSE;

Note that we use SQL standardized NULLIF control flow function. So on PHP level , never write code like : 

if ( $object->metadata->created == '0000-00-00 00:00:00')

but instead write it simpler and faster: 

if ( $object->metadata->created)

The test

MySQL Cache is off.
Query is executed 3 times every time when table is altered and I use average time.

Indexes on id and sitegroup.

* 13 rows in set (18.96 sec)

CREATE INDEX deleted_idx ON topic (metadata_deleted);

* 13 rows in set (19.00 sec)

CREATE INDEX name_idx ON topic (name);

* 13 rows in set (19.04 sec)

CREATE INDEX up_idx ON topic (up);

* 13 rows in set (19.06 sec)

CREATE INDEX metadata_created_idx ON topic (metadata_created);

* 13 rows in set (19.05 sec)

CREATE INDEX metadata_published_idx ON topic (metadata_published);

* 13 rows in set (19.12 sec)

CREATE INDEX record_extension_oid_idx ON record_extension (oid);

* 13 rows in set (19.04 sec)

CREATE INDEX record_extension_domain_idx on record_extension (domain(255));

* 13 rows in set (19.04 sec)

CREATE INDEX value_idx ON record_extension (value(255));

* 13 rows in set (19.05 sec)

CREATE INDEX name_idx ON record_extension (name(255));

* 13 rows in set (19.02 sec)

CREATE INDEX parent_guid_idx ON record_extension (parent_guid(80));

* 13 rows in set (0.01 sec) (!!)

CREATE INDEX tablename_idx ON record_extension (tablename(255));

* 13 rows in set (0.01 sec)

All indexes dropped ( except id and sitegroup ).

* 13 rows in set (19.25 sec)

Similiar query , but without parameters. 

$qb = new midgardquerybuilder("midgard_topic");

$qb->add_constraint('up', '<>', 0);

$qb->add_constraint('metadata.created', '>', '2002-02-01 00:00:00');

$qb->add_constraint('metadata.published', '>', '2002-02-01 00:00:00');

I executed this query with and without metadata indexes. The difference between queries time was none.

Conclusion

Indexes for referenced properties ( on object level and "foreign keys" on database level ) are the keys for performance. And indexes used for queries when the same table is used seem to not increase performance at all.
However this can be different for queries which join the same table.



Back

Layout Copyright © 2006 Finnish Teleservice Center Ltd Oy - Site Powered by Midgard CMS