Strange error: model saved without errors but actually DB not "UPDATE"d

Hello,

We are experiencing a very strange error. We think it is caused by a bug inside MySql/MariaDB, but I would still try to ask here hoping that someone can give me some alternative idea on which to reflect (or maybe, if it has already happened to someone ...).

I have a procedure which executes an UPDATE on a record. The procedure is executed as a job (cli).

Procedure:

public function updateElement( $id ) {
    $element = Element::findFirstById( $id );

    if( !$element ) {
        throw new App\Exceptions\NotFound();
    }

    $date = \DateTime::createFromFormat( 'Y-m-d', $element->getCurrentDate() );
    $date->modify( '+1 year' );

    $element->setCurrentDate( $date->format( 'Y-m-d' ) );

    $element->saveOrFail();
}

Note that saveOrFail() is a method that simply executes a save() on the Model and in case of failure it launches a specific exception based on our application structure.

The problem is that sometimes, apparently randomly, the new CurrentDate value is not actually saved in the database.

If we execute the same procedure on the same record, the new CurrentDate value is saved.

This procedure is performed hundreds of times a day on many different "elements". It is a procedure that is launched by a process that is always listening, which just notes that an "element" has been set in a certain way, then launches a new process on it that performs this procedure.

It may be possible to have more than one of these processes running simultaneously (each on different "elements"). We have a control that avoids to launch too many of these processes at the same time (at most 5).

After we became aware of this issue, we introduced controls in the form of log files at each point of the procedure. We have found that the "calculation" of the new date is always correct and that the Phalcon ORM always returns true on save().

We also added this at the end of the procedure:

    $element = Element::findFirstById( $id );

    if( !$element ) {
        throw new App\Exceptions\NotFound();
    }

    /* NEW */
    $oldDate = \DateTime::createFromFormat( 'Y-m-d', $element->getCurrentDate() );

    $date = \DateTime::createFromFormat( 'Y-m-d', $element->getCurrentDate() );
    $date->modify( '+1 year' );

    $element->setCurrentDate( $date->format( 'Y-m-d' ) );

    $element->saveOrFail();

    /* NEW */
    $element->refresh();

    $newDate = \DateTime::createFromFormat( 'Y-m-d', $element->getCurrentDate() );

    /* $date is the previous calculated date */
    if( $newDate == $OldDate ) {
        throw new \Exception( 'Same date' );
    }

The \Exception thrown at the end of the method is thrown only sometimes. I mean that on two procedures that actually don't update CurrentDate value, the first one is able to notice the different dates, the second one not and thinks that the two dates are different (which would mean that CurrentDate has been properly updated, when actually has not).

Otherwise, if between the time of the UPDATE and the moment of control we wait 1 minute (or in any case a few seconds), the error is always detected.

This is a very strange issue. And actually we are 99.9% sure thatbefore a certain moment it never appeared. We think it started to show up after an upgrade on our servers (maybe MariaDB?).

We use:

Nginx MySql MariaDB 10.1.37 PHP 7.0.26 Phalcon 3.3.2

Thank you for any help or ideas.

You sure you didn't updated phalcon right? I guess in other places your code didn't change? Maybe you are locking database and first update is still happening and second doesn't do update then?



10.7k

we updated Phalcon as well, from 3.1.0 to that version.

I thought about a locking issue but we don't execute the procedure under a transaction or with a forUpdate (apart from Phalcon's save implicit one). and the fact is that the date is not updated only "sometimes"...

note that we don't see any error in system logs (application server and mysql server), nginx logs, php logs.... no errors AT ALL... :(