wj&bdZddlmZddlmZddlZddlZd dZ d d dZ dZ dddZ dd Z ddd Zy)a3Precomputed evidence index. The original index handed `estimate_row` a dict of pandas DataFrames and let it do the arithmetic per target row. That cost two things: * `_debase` ran per row per pool, with `.map(lambda)` and `.apply(axis=1)` over thousands of evidence rows -- Python-level work repeated for every target that happened to share a pool. * tier 2 ran `SequenceMatcher` against EVERY live row in the market. India has 228,000 of them, so a single target row cost 228,000 fuzzy string comparisons. Profiling a 10-row schedule showed 458,062 calls and 16.6 of 18 seconds spent inside difflib. Neither is necessary. The de-based value of an evidence row does not depend on the target, and the median of a group does not either, so both are computed once here. Fuzzy team matching only ever needs to consider the DISTINCT team strings in a market -- a few thousand, not a few hundred thousand -- and only those sharing a token with the target. What `estimate_row` receives is therefore plain dicts of floats. No pandas touches the hot path. ) annotations) defaultdictNc t|}|djtj|djtj Dcic]}|j |c}j t}|djtj|djtj Dcic]}|j|c}j t}|dj|dj Dcic]}|j|c}j t}fd} |djd} | j| j Dcic] }|| | c}j t} tt|d|d} t| D cic]} | j| d | d c} tj fd | Dt| }tj"||z|z| z|zd d}|dj t|z Scc}wcc}wcc}wcc}wcc} w)avDe-base every evidence row at once. base = ama / (hour_w * weekday_w * telecast_mult * team_w * host_f) Identical arithmetic to the row-wise version, but each factor is resolved over the DISTINCT values present and then mapped, so the Python-level work is proportional to the number of distinct hours / teams / events rather than to the 5.9M rows. hourweekday telecast_typec "t|jdDcgc]#}|js|j%}}|r;tt j |Dcgc]}j |c}SdScc}wcc}w)N|g?)strsplitstripfloatnpmeanteam_w)sppartsrefs 2/var/www/html/crystal/crystal2/engine/fastindex.py team_meanz debase_column..team_mean1sh$'FLL$5CqCCAFuRWWU;cjjm;<=OCOD;sBB"B teams event_namecountryrc3(K|] }| ywN).0rhf_maps r z debase_column..9s/AfQi/s)dtypecountg& .>Nama_000)lenastypeintmapuniquehour_wto_numpyr weekday_w telecast_multfillnalistzipsethost_frfromiterclip)evrnhhwwwwttmrrtwpairsrhfdenomr!s ` @r debase_columnrBs BA F  3  # ##%f:#4#4S#9#@#@#BCaCJJqM C EEMXe_ I  c " & &&(m&:&:3&?&F&F&HICMM!  I KKS8TY? O  *,_*=*D*D*FGQC  a G IIQRWP wK  r "E U\\^<Ay|O< = F Fu MB R %r)}5 6E14U c &|j|d|jddg}t|j|dj |dj Dcic]\}}}|t |t |fc}}}Scc}}}w)z@{key: (median, count)} for one grouping, as plain Python floats.F)sortmediansize)groupbyaggr1indexr,rr()dfkeyscolgkmrs r _med_sizerP?s 4e $S)--x.@AA 8--/61C1C1E F H HgaAAa#a&! ! HH Hs%"B zevidence_index.pklcddl}ddlm}|rR||tz }|j r3 |j d5}|j |}dddid<|St|}|r ||tz }|jd}|jD cic]\}} |dk7s || } }} |j d5}|j| ||jddd|j||S|S#1swYxYw#t$rYwxYwcc} }w#1swY?xYw#t$rY|SwxYw) a&Load the prebuilt index if one exists, else build and save it. The index is a pure function of the reference snapshot, so it only has to be rebuilt when the snapshot is. Building it takes ~14s over 5.9M rows; loading it takes about a second, and every run paid the 14s before. rN)Pathrb _fuzzy_cachez.tmpwb)protocol)picklepathlibrR CACHE_NAMEexistsopenload Exceptionbuild with_suffixitemsdumpHIGHEST_PROTOCOLreplace) r cache_dirrWrRffhidxtmprNvkeeps r build_cachedrkIsB Oj ( 88: VVD\*R ++b/C*&(N#  *C Y*,A--'C%(YY[HTQA4GAqDHDH$ H2 D"v/F/F G H KKN J3J#**  I H H   J sjDDD-3D0 D.D3D0D$&D0D D DDD0$D-)D00 D=<D=c |jj}t|||d<|tj|d|ddkDz}ddlm}||ddk(j}|djdj|djdjDcic] }||| c}|d <t|gd t|d d gt|d d gd}tt}|djD]\\}}} } | |||f| <t||d<tt} |djD]\\}}} | | ||<t| |d<i} |jD]{\\}}} |j|}|s| D cgc].} |j|| s|j|| xsd|z 0}} |sYt!tj"|| ||f<}| |d<tt}t||d dk7d d gjD]\\}}} | |||<t||d<td}|jD]<\}}|D]2}|j%dD]}|s|||j'|4>|jD cic]\}} |t| c} }|d<i|d<|Scc}wcc} wcc} }w)z2Everything `estimate_row` needs, precomputed once.baserr) norm_teamsrLIVErrtkey)rrchannelrrrq)eccecccrr ec_channelsrs e_countries ec_rel_sharetkeysc ttSr)rr2rrzbuild..s K$4rzr tkey_tokensrT)evidencecopyrBrisfinitecorernr/r)r*rPrdictr`flagship_sharesharerrEr add)rr6rnliver<rgby_ec_checchriby_e_crelchansflvalstk_medtktokds rr^r^js-    Br3'BvJ BKK6 #r&zA~ 6 7B b!V+ , 1 1 3D=''+//#'=#7#7#;#B#B#DEaJqM EGDL CD i89 956 C!&HU))+! ArA !Q!hC t$FY__& Aq ! fC C!)1 A    " 8=S"1bAQ#))Ar"'a2-SS  $0CAK 1Ct$FT&\R%7 89f:MNTTV Bq " s2K;2L LLc|sy|dj|}|sy|j|}||S||f}|d}||vr||Sddlm}|jdD cgc]} | s|  } } |dj|i} t } | D]} | | j| t z} !d} | D]$}||||s||\}}| || dkDs!||f} &| ||<| Scc} w)aTeam pool for a market, exact first and fuzzy only over plausible strings. Returns (median, count) or None. Preserves the original semantics -- exact equality, else SequenceMatcher ratio >= thresh -- but compares against the market's DISTINCT team strings that share at least one token with the target, instead of every row in the market. Results are memoised, because a schedule asks the same question many times. NrxrTr) teams_matchr r|)getrrr r2)rgrrpthresh per_countryhitckcacherr<tokstokmapcandsbestcandrOr7s r match_tkeyrs! g,""7+K  //$ C  4B  E U{Ry!zz# ,!!A ,D ,   # #GR 0FE & Asu%%& D tT6 *t$DAq|q47{1v  E"I K -s C#C)r6 pd.DataFramereturnz np.ndarray)rm)rJrrrr)rr)g?)rgrrr rpr rr)__doc__ __future__r collectionsrnumpyrpandaspdrBrPrYrkr^rrrzrrs<,##1@H" B9x%rz