pdfNamer Class Reference

takes care of the correct naming of files More...

Inheritance diagram for pdfNamer:

Public Member Functions
	__construct ($pdf, $db)
	constructor for the class More...
	getTextFromPdf ($pdf)
	function executes pdftotext to extract text from file More...
	addToLog ($str)
	toDate (&$item, $key)
	converts a text string to a date. used for array walk in matchDates More...
	closestDateToToday ($dates)
	takes an array of dates and returns the closest one before today (since Paper documents have dates in the past, not in the future) More...
	cleanContent ()
	takes the PDF content and cleans it up More...
	matchDates ()
	looks regular expression dates in the content of the file More...
	replacePersonalVariables ($searchTerm)
	replaces the personal variables within a string More...
	matchSenders ()
	reads rulesets from database and executes accordingly More...
	matchPrice ()
	checks if there is a price in the text More...
	matchSubjects ()
	matching subject More...
	matchRecipients ()
	reads recipient list from database and tries to match in text More...
	addTags ()
	function adds tags once company and subject are correctly matched More...
	run ()
	main function calling relevant process steps to identify document More...
Public Member Functions inherited from ppyrd
	__construct ($db)
	constructor More...
	checkCliVsWebserver ()
	output ($string, $debug=0)
	outputs string More...

Additional Inherited Members
Public Attributes inherited from ppyrd
	$db

Detailed Description

takes care of the correct naming of files

Definition at line 16 of file ppyrd.namer.php.

Constructor & Destructor Documentation

pdfNamer::__construct	(		$pdf,
			$db
	)

constructor for the class

Parameters

string	$pdf	with file name to process
string	$db	with database connection

Returns

none

Bug:

this still checks for a date at the start - but this may change according to configured value.

Definition at line 24 of file ppyrd.namer.php.

    {
      // cleaning the log
      $this->log = "";

      // dont output debug information
      $this->debug = false;

      // creating db handler to talk to DB
      $this->db=$db;

      // old name equals new name in the beginning
      $this->oldName=$pdf;

      // set new name only if it has not been applied already (e.g. a document is not fully recognized and rematched with updated DB entries)
      if (!preg_match('(ddatum|ffirma|bbetreff|wwer|bbetrag)',$pdf) && !preg_match('(^\d{8}\s\-)',$pdf)) {
          // getting new file name structure from database
          $this->newName=$this->db->getConfigValue('newFilenameStructure');
          // appending old file name - otherwise just add .pdf to the end
          if ($db->getConfigValue('appendOldFilename')==1) {
            $this->newName .=  $pdf;
          } else {
            $this->newName .= ".pdf";
          }


          }
        else {
          $this->newName=$pdf;
        }

      $this->companyName = "";
      $this->subjectName = "";

      // standard tag if no tags are found
      $this->tags = "[nt]";



      // what mimimum score is required until we accept the company as correct
      $this->companyMatchRating = $this->db->getConfigValue("companyMatchRating");

      // what mimimum score is required until we accept the company as correct
      $this->subjectMatchRating = $this->db->getConfigValue("subjectMatchRating");

      $this->dateRegEx = $this->db->getConfigValue("dateRegEx");

      // reads the pdf
      $this->getTextFromPdf($pdf);
    }

Member Function Documentation

pdfNamer::addTags

(

)

function adds tags once company and subject are correctly matched

Definition at line 464 of file ppyrd.namer.php.

                       {
      // tossing all tags into one array
      @$alltags = array_merge($this->matchedCompanyTags, $this->matchedSubjectTags);

      // splitting up comma separated values and putting them back into the array
      @$tags = explode(',',join(",", $alltags));

      // cleaning the tags
      $cleantags = "";
      foreach ($tags as $tag) {
        $tag = trim($tag);
        if (!empty($tag))
          $cleantags[] = "[$tag]";
      }

      // removing duplicates
      if (is_array($cleantags)) {
        $cleantags = array_unique($cleantags);

        // sorting tags
        asort($cleantags);

        // joining them into one string
        $this->tags=implode($cleantags);

      $this->output("tags:    " . $this->tags);
      }
        else {
          $this->output("tags:    no tags to assign");
        }

      // changing date in fileName only if tags are to assign
      if (!empty($this->tags))
        $this->newName = str_replace("[nt]",$this->tags, $this->newName);

    }

pdfNamer::addToLog

(

$str

)

adds a string to a log message which later can be written to DB out to STDOUT

Parameters

string

$str

contains the message which shall be added to log

Returns

none

Definition at line 93 of file ppyrd.namer.php.

                            {
      $this->log .= $str . "\n";
    }

pdfNamer::cleanContent

(

)

takes the PDF content and cleans it up

Parameters

none

Returns

none

Definition at line 137 of file ppyrd.namer.php.

    {
      // get everything into one long string
      $this->content = implode(" ", $this->content);

      // convert everything to lowercase to avoid case sensitive mismatches
      $this->content = strtolower($this->content);

      // todo: remove everything but digits and letters
      $this->content = preg_replace($this->db->getConfigValue('stripCharactersFromContent'), " ", $this->content);

      // remove spaces if there is more than one (double space, tripple space etc.);
      $this->content = preg_replace("/\s\s+/", " ", $this->content);

      //var_dump($this->content);
    }

pdfNamer::closestDateToToday

(

$dates

)

takes an array of dates and returns the closest one before today (since Paper documents have dates in the past, not in the future)

Parameters

array

$dates

containing all dates in YYYYMMDD format

Returns

string YYYYMMDD if match or ddatum if failed to match a date

Definition at line 121 of file ppyrd.namer.php.

    {
      arsort($dates);
      foreach ($dates as $date) {
        if ($date<=date('Ymd'))
          return $date;
      }
      return "ddatum";
    }

pdfNamer::getTextFromPdf

(

$pdf

)

function executes pdftotext to extract text from file

Parameters

string

$pdf

name of file

Definition at line 81 of file ppyrd.namer.php.

                                  {
      // reads content into $this->content
      exec('pdftotext -layout "' . $pdf . '" -', $this->content);
      //exec('pdftotext "' . $pdf . '"');
    }

pdfNamer::matchDates

(

)

looks regular expression dates in the content of the file

Parameters

none

Returns

none

Definition at line 160 of file ppyrd.namer.php.

    {
      $this->addToLog('');
      $this->addToLog('===');
      $this->addToLog('LOOKING FOR DATES');

      // Datumsformate
      preg_match_all ($this->dateRegEx, $this->content, $dates);

      // only consider full matches and remove duplicates
      $dates = array_unique($dates[0]);

      // getting into YYYYmmdd format
      array_walk($dates, 'self::toDate');
      $dates = array_unique($dates);

      // most likely date found
      $this->newDate = $this->closestDateToToday($dates);

      // changing date in fileName
      $this->newName = str_replace("ddatum",$this->newDate, $this->newName);

    }

pdfNamer::matchPrice

(

)

checks if there is a price in the text

Definition at line 303 of file ppyrd.namer.php.

    {
      // matching all potential price mentions
      preg_match_all($this->db->getConfigValue('matchPriceRegex'), $this->content, $results);

      // getting values of full match only
      $prices = array_values($results[0]);

      // removing all non numeric characters except comma and period
      $prices = preg_replace("/[^0-9,.]/", "", $prices);
      $maxprice = 0;
      foreach ($prices as $price) {
        $price = floatval(str_replace(',','.',str_replace('.','', $price)));
        if ($price > $maxprice) $maxprice = $price;
      }

      // setting max price
      $this->price=number_format($maxprice,2,",",".");

      $this->newName = str_replace("bbetrag","EUR".$this->price, $this->newName);

      $this->output("amount:  EUR" . $this->price);

      }

pdfNamer::matchRecipients

(

)

reads recipient list from database and tries to match in text

Parameters

none

Returns

none

Definition at line 432 of file ppyrd.namer.php.

                               {
      // looking for active rules from database to check document against
      $results = $this->db->getActiveRecipients();
      $recipients = array();

      // for each rule check if the name occures in the text.
      while ($row = $results->fetchArray()) {
        $cfound = substr_count($this->content, strtolower($row['recipientName']));
        $this->output("look for " . $row['recipientName'] . " found $cfound", 1);
        @$recipients[$row['shortNameForFile']] += $cfound;
      }

      // sort the results alphabetically
      asort($recipients);

      // kill all entries which have not been matched
      foreach ($recipients as $name => $score) {
        if ($score == 0) unset($recipients[$name]);
      }

      // switch key & values => as we want to have the name and not the # of hits
      // join all hits with a comma
      $recipients = implode(',',array_flip($recipients));

      // write the new name
      if (!empty($recipients))
        $this->newName = str_replace("wwer",$recipients, $this->newName);
    }

pdfNamer::matchSenders

(

)

reads rulesets from database and executes accordingly

Parameters

none

Returns

none

Definition at line 210 of file ppyrd.namer.php.

    {
      // looking for active rules from database to check document against
      $results = $this->db->getActiveSenders();
      $company = array();
      $tmpMatchedCompanyTags= array();

      // start  matching search terms vs content
      while ($row = $results->fetchArray()) {

        // checking if there are multiple search terms separated by a comma

        // start - just one searchterm
        if (strpos($row['foundWords'], ",")=== false) {

          $searchTerms = strtolower($this->replacePersonalVariables($row['foundWords']));

          // checking if we found it at least once
          if (substr_count($this->content, $searchTerms)>0) {
            @$company[$row['fileCompany']] += $row['companyScore'];
            // keeping a list of match hits for later tagging
            $tmpMatchedCompanyTags[$row['fileCompany']][]=$row['tags'];

            }
        } // end - just one search


        // start - multiple search terms
         else {

          // separating search terms and removing white spaces
          $split = explode(',',  strtolower($row['foundWords']));

          // break variable to stop in case one word was not found
          $foundAll = true;
          foreach ($split as $value) {
            if($foundAll) {

              // removing any whitespace
              $value = trim($this->replacePersonalVariables($value));

              // counting occurances
              $cfound = substr_count($this->content, $value);


              // setting stop variable since nothing was found
              if ($cfound==0) {
                $foundAll= false;
              }
            }
          }

          // found all - lets write the result
          if ($foundAll)    {
            @$company[$row['fileCompany']] += $row['companyScore'];
            // keeping a list of match hits for later tagging
            $tmpMatchedCompanyTags[$row['fileCompany']][]=$row['tags'];

            // writing log
            $this->addToLog('"' . $row['foundWords'] . '" ' . " found - " . $row['companyScore'] . " points for company " . $row['fileCompany']);
          }

          // not all found - thus no results to write
          else {
          }
        }

      } // end - matching search terms vs content


      // sorting so highest match is on top
      arsort($company);

      if (isset($company[key($company)])) {
        $companyMatchRating = $company[key($company)];
        $this->companyName = key($company);
        $this->matchedCompanyTags = $tmpMatchedCompanyTags[$this->companyName];

        // checking match ranking
        $this->output("company: " . $this->companyName . " scored " . $companyMatchRating);

        if ($companyMatchRating >= $this->companyMatchRating) {
          $this->newName = str_replace("ffirma",$this->companyName, $this->newName);
        }
      }



    }

pdfNamer::matchSubjects

(

)

matching subject

Definition at line 332 of file ppyrd.namer.php.

                             {
      // looking for active rules from database to check document against
      $results = $this->db->getActiveSubjects();
      $subject = array();

      $tmpMatchedSubjectTags = array();

      // start  matching search terms vs content
      while ($row = $results->fetchArray()) {

        // checking if the found company matches the company specified in subject rule
        // also checking that it is not empty
        @$tmpFoundCompany = trim($row['foundCompany']);
        if ($tmpFoundCompany== $this->companyName || empty($tmpFoundCompany)) {


          // checking if there are multiple search terms separated by a comma

          // start - just one searchterm
          if (strpos($row['foundWords'], ",")=== false) {

            // checking if we found it at least once
            $searchTerm = strtolower($this->replacePersonalVariables($row['foundWords']));
            if (substr_count($this->content, $searchTerm)>0) {
              @$subject[$row['fileSubject']] += $row['subjectScore'];

              // keeping a list of match hits for later tagging
              $tmpMatchedSubjectTags[$row['fileSubject']][]=$row['tags'];
              }
          } // end - just one search


          // start - multiple search terms
           else {

            // separating search terms and removing white spaces
            $split = explode(',',  strtolower($row['foundWords']));

            // break variable to stop in case one word was not found
            $foundAll = true;
            foreach ($split as $value) {
              if($foundAll) {

                // removing any whitespace
                $value = trim($this->replacePersonalVariables($value));

                // counting occurances
                $cfound = substr_count($this->content, $value);


                // setting stop variable since nothing was found
                if ($cfound==0) {
                  $foundAll= false;
                }
              }
            }

            // found all - lets write the result
            if ($foundAll)  {
              @$subject[$row['fileSubject']] += $row['subjectScore'];
              // keeping a list of match hits for later tagging
              $tmpMatchedSubjectTags[$row['fileSubject']][]=$row['tags'];
              // writing log
              $this->addToLog('"' . $row['foundWords'] . '" ' . " found - " . $row['subjectScore'] . " points for subject " . $row['fileSubject']);
            }

            // not all found - thus no results to write
            else {
              }
            }
          } // end check if company name matches
      } // end - matching search terms vs content


      // sorting so highest match is on top
      arsort($subject);

      @$subjectMatchRating = $subject[key($subject)];
      $this->subjectName = key($subject);
      @$this->matchedSubjectTags = $tmpMatchedSubjectTags[$this->subjectName];

      // checking match ranking
      $this->output("subject: " . $this->subjectName . " scored " . $subjectMatchRating);

      if ($subjectMatchRating >= $this->subjectMatchRating) {
        $this->newName = str_replace("bbetreff",$this->subjectName, $this->newName);
      }




    }

pdfNamer::replacePersonalVariables

(

$searchTerm

)

replaces the personal variables within a string

Parameters

string

$searchTerm

Definition at line 189 of file ppyrd.namer.php.

    {
      // getting all replacement terms
      $results = $this->db->getPersonalVariables();

      // looping string thru all terms
      while ($row = $results->fetchArray()) {
          $searchTerm= str_replace($row['variableName'], $row['replaceWith'], $searchTerm);
      }


      // done
      return $searchTerm;
    }

pdfNamer::run

(

)

main function calling relevant process steps to identify document

Parameters

none

Returns

none

Definition at line 508 of file ppyrd.namer.php.

                   {
      // cleaning content of the PDF document
      $this->cleanContent();

      // looking for dates in content
      $this->matchDates();

      // matching rule sets from database
      //$this->matchRules();

      // match recipients from database
      $this->matchSenders();
      // match recipients from database
      $this->matchSubjects();

      // matching tags once company and subject are matched
      $this->addTags();



      // match recipients from database
      $this->matchRecipients();

      //
      $this->matchPrice();

      // renaming the file in case everything matched
      if (!preg_match('(ddatum|ffirma|bbetreff|wwer|bbetrag)',$this->newName)) {
          exec('mv --backup=numbered "' . $this->oldName . '" "../outbox/' . $this->newName . '"');
          }
        else {
          // dont move in case something is still unmatched
          if ($this->oldName != $this->newName) {
            exec('mv --backup=numbered "' . $this->oldName . '" "' . $this->newName . '"');
          }
        }

      $this->output("new name: " . $this->newName);


      // logging everything to database
      $this->db->writeLog($this->oldName, $this->newName, $this->content, $this->log);
    }

pdfNamer::toDate	(	&	$item,
			$key
	)

converts a text string to a date. used for array walk in matchDates

Parameters

pointer	$item	pointer to array item
string	$key	array key name

Returns

none

Definition at line 104 of file ppyrd.namer.php.

                                    {
      self::addtolog("Date found $item");
      if (strpos($item, "im") !== false) {
        $item =str_replace("im ","",$item);
        $item = date("Ymt", strtotime($item));
      } else {
        $item = date("Ymd", strtotime($item));
      }
    }

---

The documentation for this class was generated from the following file:

• paperyard/paperyard/backend/ppyrd.namer.php