The development of cleat fractures in coal rocks leads to reduced strength, and the influence of drilling fluids during drilling further escalates the risk of borehole instability. By combining laboratory experiments, theoretical analysis, and numerical simulations, this study elucidates the mechanisms of instability in cleated coal rocks under the influence of drilling fluids: (1)the clay minerals in the coal rock are predominantly composed of kaolinite, which is resistant to hydration, with little to no montmorillonite present, and a low proportion of illite/smectite(I/S) mixed layers. Consequently, both hydration expansion (averaging 0.35%) and rolling dispersion (averaging 89.64%) are minimal, indicating the instability of coal rocks is primarily driven by mechanical factors. (2) Vertically intersecting face and end cleats create flow channels that allow drilling fluid to intrude into the coal strata. Since the dimensions of face cleats are generally larger than those of end cleats, face cleats are more susceptible to fluid intrusion, leading to borehole instability. (3) The intrusion of drilling fluid into the cleat fractures leads to an increase in formation pressure around the well and a reduction in radial stress, thereby raising the risk of borehole instability. Notably, for high-permeability face cleats and cross-cutting cleats that intersect the boreholes, deeper fluid intrusion further elevates formation pressure and diminishes radial stress, exacerbating the risk of instability. (4) Additionally, the characteristics of the cleats also affect the depth to which drilling fluid intrudes into the coal strata. In cleats with greater widths and higher densities, the drilling fluid penetrates deeper and generates higher pressures near the borehole wall, thereby increasing the likelihood of instability. Therefore, plugging particles should be added into the drilling fluid according to the size of the cleats and the density of the drilling fluid should be maintained within a reasonable range, in order to minimize borehole instability caused by fluid intrusion. The study provides a new perspective to understand the instability mechanism of cleats under the action of drilling fluid, and provides theoretical guidance for the analysis of coal rock wall stability.