The Lagrangian dynamical models describing the evolution of the fine-scale and coarse-grained Velocity Gradient Tensor (VGT) are developed under the Physics-Informed Machine Learning (PIML) framework. The pressure hessian contribution is re-constructed using the integrity bases and invariants of VGT, which provides an improved representation of its magnitude and orientation. The unclosed incoherent small scale fluctuations are modeled using ML techniques trained on Lagrangian data from a high-Reynolds number Direct Numerical Simulation (DNS). Certain constraints, such as Galilean invariance, rotational invariance, and zero-pressure work condition, are enforced to implement known physics into the ML model. Then, a comprehensive diagnostic test is performed. Statistics of the flow, as indicated by the joint PDF of second and third invariants of the VGT, show good agreement with the ground-truth DNS. Some important features regarding the structure of the turbulence are correctly reproduced by the model including skewed distribution of the velocity gradient, vorticity-strain rate alignment and vortex stretching mechanism. Future work will be focused on extending the Lagrangian Tetrad framework to include the geometry information in the PIML model of the coarse-grained VGT.