RIP Philo

Best PHP profiler?

A customer has given me a dataset to import that is 100x bigger than the scripts were designed to handle, and they're not handling it well enough, execution time is too long.
Permalink ,ndo 
July 5th, 2018 8:42pm
Print statements
try with small sets. See what the scaling is
Data size in G?
Machine size?
Where stored , where is output written to
Permalink X 
July 5th, 2018 8:52pm

heh I just finished reading that one before coming back here
Permalink ,ndo 
July 5th, 2018 8:56pm

heh I just finished reading that one before coming back here

except I found it at
Permalink ,ndo 
July 5th, 2018 8:57pm
How complex is the logic?  What database?  Is it SQL or that noSQL trash?

Load the data into the database, then write SQL to clean/load it to the destination tables.

Or just rent an AWS large instance and leave it running for a few days.
Permalink Send private email FSK 
July 5th, 2018 9:13pm
>noSQL trash?


You sound unhinged

This is why no one listens to you

This is why you fail most interviews
Permalink Public service 
July 5th, 2018 9:25pm
Upgrading to php 7 is much more efficient and faster.

Use stored procedures in the database and prepared statements

Use pdo and the latest matching drivers

Use indexes

Don’t use the default index

Use fetch all instead of single fetch with a loop
Permalink Paul Daniels 
July 6th, 2018 1:06am
There was a podcast on PHP Roundtable recently on static analysis:
Permalink Grumpy Old Git 
July 6th, 2018 1:22am
My public servers are v5.6 and there's no pressing reason to change them at this moment.

I've always been partial to static analysis and will check out the podcast when I get a chance.
Permalink ,ndo 
July 6th, 2018 2:04am
You'd be helping to save the planet!

Rasmus Lerdorf - Deploying PHP 7:
Permalink YouTubeBot 
July 6th, 2018 2:22am
Well I installed xhprof and that has given me a couple of useful places to start.
Permalink ,ndo 
July 6th, 2018 6:44am
Keep us posted
Permalink Grumpy Old Git 
July 6th, 2018 6:52am
Pushing into a database ?

Perhaps the inserts are poorly designed, and the profiler might not catch that.

Be sure to loop over prepared statements.
Permalink John 
July 6th, 2018 7:34am
Basically commit and execute. Lookups are repeated (what happens with new rows depends on the contents of old rows), which wasn't a problem with small imports but the large import yields a much higher redundancy. I'm going to have to cache the results from the lookups.
Permalink ,ndo 
July 6th, 2018 7:38am
yep, I always use prepared statements
Permalink ,ndo 
July 6th, 2018 7:40am

This topic is archived. No further replies will be accepted.

Other topics: July, 2018 Other topics: July, 2018 Recent topics Recent topics